2024-09-10T20:13:09.106 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-10T20:13:09.125 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T20:13:09.217 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561 branch: wip-bharath7-testing-2024-09-10-1409 description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} email: skanta@redhat.com first_in_suite: false job_id: '7899561' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-bharath7-testing-2024-09-10-1409 ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 5 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2 random: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(TOO_FEW_PGS\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-bharath7-testing-2024-09-10-1409 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 owner: scheduled_skanta@teuthology priority: 99 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 5245 sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-bharath7-testing-2024-09-10-1409 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 targets: smithi153.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIDRdRZxOj9w6XChkKnhcfza9lnOD4JcUM6bRzevopJ3cfPbdQ1DGMIVYw1yfIlmUXMauFz33T5tQDtogNZZBso= smithi176.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLKp1W2il4oXQDZt3QK2Nr5PTBtX+3C2ODJBk5ZDJPKgmIY6LIyZdiyriLTj7X8jBRz//2/wytqMBVD/qaj5O0U= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 store_thrash: true thrash_delay: 1 thrash_many: true - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_15:30:27 tube: smithi user: skanta verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-10T20:13:09.218 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa; will attempt to use it 2024-09-10T20:13:09.218 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa/tasks 2024-09-10T20:13:09.219 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-10T20:13:09.220 INFO:teuthology.task.internal:Checking packages... 2024-09-10T20:13:09.243 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46' 2024-09-10T20:13:09.243 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-10T20:13:09.243 INFO:teuthology.packaging:ref: None 2024-09-10T20:13:09.243 INFO:teuthology.packaging:tag: None 2024-09-10T20:13:09.243 INFO:teuthology.packaging:branch: wip-bharath7-testing-2024-09-10-1409 2024-09-10T20:13:09.243 INFO:teuthology.packaging:sha1: bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:13:09.243 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-bharath7-testing-2024-09-10-1409 2024-09-10T20:13:09.383 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:13:09.384 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-10T20:13:09.391 INFO:teuthology.task.internal:no buildpackages task found 2024-09-10T20:13:09.392 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-10T20:13:09.445 INFO:teuthology.task.internal:Saving configuration 2024-09-10T20:13:09.459 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-10T20:13:09.485 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-10T20:13:09.504 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi153.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 20:07:35.045570', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDj5C+r5jcYpcLo+zeyjGImb40SphkcqwExGcjlVOYRO'} 2024-09-10T20:13:09.520 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi176.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-10 20:07:35.047138', 'locked_by': 'scheduled_skanta@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMgo1s2b9QU3UKADfZgGqbJCy3+z/xgCc1WI5QVsyDra'} 2024-09-10T20:13:09.521 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-10T20:13:09.527 INFO:teuthology.task.internal:roles: ubuntu@smithi153.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-10T20:13:09.527 INFO:teuthology.task.internal:roles: ubuntu@smithi176.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-10T20:13:09.528 INFO:teuthology.run_tasks:Running task console_log... 2024-09-10T20:13:09.596 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f1640f2bbe0>, signals=[15]) 2024-09-10T20:13:09.596 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-10T20:13:09.601 INFO:teuthology.task.internal:Opening connections... 2024-09-10T20:13:09.602 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi153.front.sepia.ceph.com 2024-09-10T20:13:09.603 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:13:09.686 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi176.front.sepia.ceph.com 2024-09-10T20:13:09.687 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:13:09.768 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-10T20:13:09.774 DEBUG:teuthology.orchestra.run.smithi153:> uname -m 2024-09-10T20:13:09.779 INFO:teuthology.orchestra.run.smithi153.stdout:x86_64 2024-09-10T20:13:09.779 DEBUG:teuthology.orchestra.run.smithi153:> cat /etc/os-release 2024-09-10T20:13:09.827 INFO:teuthology.orchestra.run.smithi153.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-10T20:13:09.827 INFO:teuthology.orchestra.run.smithi153.stdout:NAME="Ubuntu" 2024-09-10T20:13:09.827 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_ID="22.04" 2024-09-10T20:13:09.828 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-10T20:13:09.828 INFO:teuthology.orchestra.run.smithi153.stdout:VERSION_CODENAME=jammy 2024-09-10T20:13:09.828 INFO:teuthology.orchestra.run.smithi153.stdout:ID=ubuntu 2024-09-10T20:13:09.828 INFO:teuthology.orchestra.run.smithi153.stdout:ID_LIKE=debian 2024-09-10T20:13:09.828 INFO:teuthology.orchestra.run.smithi153.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-10T20:13:09.828 INFO:teuthology.orchestra.run.smithi153.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-10T20:13:09.828 INFO:teuthology.orchestra.run.smithi153.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-10T20:13:09.828 INFO:teuthology.orchestra.run.smithi153.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-10T20:13:09.828 INFO:teuthology.orchestra.run.smithi153.stdout:UBUNTU_CODENAME=jammy 2024-09-10T20:13:09.829 INFO:teuthology.lock.ops:Updating smithi153.front.sepia.ceph.com on lock server 2024-09-10T20:13:09.851 DEBUG:teuthology.orchestra.run.smithi176:> uname -m 2024-09-10T20:13:09.856 INFO:teuthology.orchestra.run.smithi176.stdout:x86_64 2024-09-10T20:13:09.856 DEBUG:teuthology.orchestra.run.smithi176:> cat /etc/os-release 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:NAME="Ubuntu" 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_ID="22.04" 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:VERSION_CODENAME=jammy 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:ID=ubuntu 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:ID_LIKE=debian 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-10T20:13:09.902 INFO:teuthology.orchestra.run.smithi176.stdout:UBUNTU_CODENAME=jammy 2024-09-10T20:13:09.903 INFO:teuthology.lock.ops:Updating smithi176.front.sepia.ceph.com on lock server 2024-09-10T20:13:09.926 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-10T20:13:09.935 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-10T20:13:09.942 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-10T20:13:09.942 DEBUG:teuthology.orchestra.run.smithi153:> test '!' -e /home/ubuntu/cephtest 2024-09-10T20:13:09.944 DEBUG:teuthology.orchestra.run.smithi176:> test '!' -e /home/ubuntu/cephtest 2024-09-10T20:13:09.949 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-10T20:13:09.955 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-10T20:13:09.955 DEBUG:teuthology.orchestra.run.smithi153:> test -z $(ls -A /var/lib/ceph) 2024-09-10T20:13:09.989 DEBUG:teuthology.orchestra.run.smithi176:> test -z $(ls -A /var/lib/ceph) 2024-09-10T20:13:09.996 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-10T20:13:10.047 INFO:teuthology.run_tasks:Running task kernel... 2024-09-10T20:13:10.065 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-10T20:13:10.065 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-10T20:13:10.065 DEBUG:teuthology.orchestra.run.smithi153:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T20:13:10.066 DEBUG:teuthology.orchestra.run.smithi176:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-10T20:13:10.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:13:10.070 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-10T20:13:10.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:13:10.071 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-09-10T20:13:10.116 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-56-generic 2024-09-10T20:13:10.117 INFO:teuthology.task.kernel:Running kernel on smithi153: 5.15.0-56-generic 2024-09-10T20:13:10.117 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2024-09-10T20:13:10.124 INFO:teuthology.orchestra.run.smithi176.stdout:5.15.0-56-generic 2024-09-10T20:13:10.124 INFO:teuthology.task.kernel:Running kernel on smithi176: 5.15.0-56-generic 2024-09-10T20:13:10.125 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get clean 2024-09-10T20:13:10.253 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2024-09-10T20:13:10.261 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2024-09-10T20:13:10.523 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-10T20:13:10.523 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-10T20:13:10.546 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-10T20:13:10.549 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-10T20:13:10.592 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-10T20:13:10.600 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-10T20:13:10.947 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-10T20:13:10.954 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-10T20:13:10.974 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-10T20:13:10.974 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-10T20:13:11.086 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-10T20:13:11.093 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-10T20:13:11.269 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-10T20:13:11.286 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,987 kB] 2024-09-10T20:13:11.290 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,987 kB] 2024-09-10T20:13:11.391 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-10T20:13:11.408 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-10T20:13:11.416 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-10T20:13:11.417 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-10T20:13:11.437 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-10T20:13:11.441 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-10T20:13:11.442 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-10T20:13:11.451 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-10T20:13:11.452 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-10T20:13:11.453 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-10T20:13:11.454 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-10T20:13:11.455 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-10T20:13:11.456 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-10T20:13:11.457 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-10T20:13:11.458 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-10T20:13:11.513 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-10T20:13:11.514 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-10T20:13:11.518 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-10T20:13:11.535 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-10T20:13:11.535 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-10T20:13:11.535 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-10T20:13:11.542 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-10T20:13:11.542 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,122 kB] 2024-09-10T20:13:11.548 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-10T20:13:11.569 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-10T20:13:11.569 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-10T20:13:11.569 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-10T20:13:11.570 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-10T20:13:11.572 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-10T20:13:11.575 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-10T20:13:11.576 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-10T20:13:11.576 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-10T20:13:11.577 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-10T20:13:11.577 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-10T20:13:11.577 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-10T20:13:11.589 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-10T20:13:11.590 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,122 kB] 2024-09-10T20:13:11.594 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-10T20:13:11.595 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-10T20:13:11.603 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-10T20:13:11.604 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-10T20:13:11.609 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-10T20:13:11.613 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-10T20:13:11.614 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-10T20:13:11.614 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-10T20:13:11.615 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-10T20:13:11.615 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-10T20:13:11.617 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-10T20:13:11.618 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-10T20:13:11.618 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-10T20:13:11.619 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-10T20:13:11.656 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-10T20:13:11.657 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-10T20:13:11.674 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-10T20:13:11.674 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-10T20:13:11.675 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-10T20:13:11.676 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-10T20:13:11.677 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-10T20:13:11.684 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-10T20:13:11.684 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-10T20:13:11.685 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-10T20:13:11.686 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-10T20:13:11.688 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-10T20:13:11.688 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-10T20:13:11.689 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-10T20:13:11.690 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-10T20:13:11.766 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-10T20:13:11.766 INFO:teuthology.orchestra.run.smithi176.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-10T20:13:11.770 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-10T20:13:11.770 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-10T20:13:11.770 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-10T20:13:11.771 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-10T20:13:11.771 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-10T20:13:17.687 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 15.8 MB in 3s (5,868 kB/s) 2024-09-10T20:13:17.722 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 15.8 MB in 3s (5,833 kB/s) 2024-09-10T20:13:19.083 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-09-10T20:13:19.097 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-10T20:13:19.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T20:13:19.116 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T20:13:19.153 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-09-10T20:13:19.171 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-10T20:13:19.338 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2024-09-10T20:13:19.339 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2024-09-10T20:13:19.356 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-09-10T20:13:19.357 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-09-10T20:13:19.503 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T20:13:19.503 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-10T20:13:19.503 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-10T20:13:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-10T20:13:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-10T20:13:19.504 INFO:teuthology.orchestra.run.smithi153.stdout: python2.7-minimal 2024-09-10T20:13:19.504 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T20:13:19.505 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2024-09-10T20:13:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-10T20:13:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-10T20:13:19.505 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-10T20:13:19.507 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2024-09-10T20:13:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-10T20:13:19.507 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2024-09-10T20:13:19.507 INFO:teuthology.orchestra.run.smithi153.stdout: thermald 2024-09-10T20:13:19.522 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T20:13:19.522 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-10T20:13:19.522 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-10T20:13:19.523 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-10T20:13:19.523 INFO:teuthology.orchestra.run.smithi176.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-10T20:13:19.523 INFO:teuthology.orchestra.run.smithi176.stdout: python2.7-minimal 2024-09-10T20:13:19.523 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T20:13:19.524 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2024-09-10T20:13:19.524 INFO:teuthology.orchestra.run.smithi176.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-10T20:13:19.525 INFO:teuthology.orchestra.run.smithi176.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-10T20:13:19.525 INFO:teuthology.orchestra.run.smithi176.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-10T20:13:19.526 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2024-09-10T20:13:19.526 INFO:teuthology.orchestra.run.smithi176.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-10T20:13:19.526 INFO:teuthology.orchestra.run.smithi176.stdout:Recommended packages: 2024-09-10T20:13:19.526 INFO:teuthology.orchestra.run.smithi176.stdout: thermald 2024-09-10T20:13:19.547 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2024-09-10T20:13:19.547 INFO:teuthology.orchestra.run.smithi153.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-10T20:13:19.548 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-10T20:13:19.548 INFO:teuthology.orchestra.run.smithi153.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-10T20:13:19.549 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages will be upgraded: 2024-09-10T20:13:19.549 INFO:teuthology.orchestra.run.smithi153.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-10T20:13:19.569 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2024-09-10T20:13:19.569 INFO:teuthology.orchestra.run.smithi176.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-10T20:13:19.569 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-10T20:13:19.569 INFO:teuthology.orchestra.run.smithi176.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-10T20:13:19.571 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages will be upgraded: 2024-09-10T20:13:19.571 INFO:teuthology.orchestra.run.smithi176.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-10T20:13:19.730 INFO:teuthology.orchestra.run.smithi153.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-10T20:13:19.730 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 113 MB of archives. 2024-09-10T20:13:19.730 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-10T20:13:19.730 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-10T20:13:19.750 INFO:teuthology.orchestra.run.smithi176.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-10T20:13:19.751 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 113 MB of archives. 2024-09-10T20:13:19.751 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-10T20:13:19.751 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-10T20:13:21.093 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-10T20:13:21.228 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-10T20:13:21.486 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-10T20:13:21.577 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-10T20:13:23.627 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-10T20:13:23.627 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-10T20:13:23.627 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-10T20:13:23.804 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-10T20:13:23.804 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-10T20:13:23.805 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-10T20:13:23.965 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-10T20:13:24.044 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-10T20:13:24.147 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-10T20:13:24.227 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-10T20:13:24.439 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 113 MB in 4s (25.3 MB/s) 2024-09-10T20:13:24.617 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-10T20:13:24.624 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 113 MB in 5s (24.4 MB/s) 2024-09-10T20:13:24.790 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-10T20:13:26.728 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-10T20:13:26.732 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T20:13:26.764 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-10T20:13:26.768 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T20:13:26.812 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:27.089 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:29.104 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-10T20:13:29.125 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T20:13:29.215 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:29.324 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-10T20:13:29.345 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T20:13:29.439 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:29.910 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-10T20:13:29.931 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T20:13:29.977 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:29.992 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-10T20:13:30.011 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T20:13:30.048 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:36.388 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-10T20:13:36.428 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-10T20:13:36.528 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-10T20:13:36.555 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-10T20:13:36.879 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-10T20:13:36.897 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-10T20:13:37.017 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-10T20:13:37.026 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-10T20:13:37.328 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-10T20:13:37.351 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-10T20:13:37.395 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-10T20:13:37.409 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-10T20:13:37.432 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-10T20:13:37.715 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-10T20:13:42.705 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-10T20:13:42.742 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T20:13:42.780 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:43.378 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-10T20:13:43.416 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-10T20:13:43.453 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:45.859 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-10T20:13:46.916 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-10T20:13:47.215 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-10T20:13:47.216 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-10T20:13:47.345 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:47.363 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-10T20:13:47.463 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-10T20:13:47.567 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:47.721 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-10T20:13:47.843 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:47.969 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-10T20:13:48.123 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:48.869 INFO:teuthology.orchestra.run.smithi176.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-10T20:13:48.869 INFO:teuthology.orchestra.run.smithi176.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-10T20:13:48.870 INFO:teuthology.orchestra.run.smithi176.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-10T20:13:48.870 INFO:teuthology.orchestra.run.smithi176.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-10T20:13:49.043 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:49.457 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-10T20:13:49.457 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-10T20:13:49.457 INFO:teuthology.orchestra.run.smithi153.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-10T20:13:49.458 INFO:teuthology.orchestra.run.smithi153.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-10T20:13:49.643 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:50.469 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:51.060 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:51.871 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-10T20:13:51.971 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-10T20:13:52.848 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:52.862 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-10T20:13:52.938 INFO:teuthology.orchestra.run.smithi176.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-10T20:13:52.938 INFO:teuthology.orchestra.run.smithi176.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-10T20:13:52.988 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-10T20:13:53.156 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-10T20:13:53.381 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-10T20:13:53.381 INFO:teuthology.orchestra.run.smithi153.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-10T20:14:06.943 INFO:teuthology.orchestra.run.smithi176.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-10T20:14:06.944 INFO:teuthology.orchestra.run.smithi176.stdout:Sourcing file `/etc/default/grub' 2024-09-10T20:14:06.961 INFO:teuthology.orchestra.run.smithi176.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T20:14:06.977 INFO:teuthology.orchestra.run.smithi176.stdout:Generating grub configuration file ... 2024-09-10T20:14:07.146 INFO:teuthology.orchestra.run.smithi153.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-10T20:14:07.146 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub' 2024-09-10T20:14:07.173 INFO:teuthology.orchestra.run.smithi153.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T20:14:07.199 INFO:teuthology.orchestra.run.smithi153.stdout:Generating grub configuration file ... 2024-09-10T20:14:07.245 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-10T20:14:07.267 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-10T20:14:07.501 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-10T20:14:07.528 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-10T20:14:07.560 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-10T20:14:07.566 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-10T20:14:07.631 INFO:teuthology.orchestra.run.smithi176.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-10T20:14:07.636 INFO:teuthology.orchestra.run.smithi176.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-10T20:14:07.787 INFO:teuthology.orchestra.run.smithi176.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-10T20:14:07.788 INFO:teuthology.orchestra.run.smithi176.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-10T20:14:07.788 INFO:teuthology.orchestra.run.smithi176.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-10T20:14:07.829 INFO:teuthology.orchestra.run.smithi176.stdout:done 2024-09-10T20:14:07.843 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-10T20:14:07.850 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-10T20:14:07.929 INFO:teuthology.orchestra.run.smithi153.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-10T20:14:07.935 INFO:teuthology.orchestra.run.smithi153.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-10T20:14:08.100 INFO:teuthology.orchestra.run.smithi153.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-10T20:14:08.100 INFO:teuthology.orchestra.run.smithi153.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-10T20:14:08.100 INFO:teuthology.orchestra.run.smithi153.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-10T20:14:08.139 INFO:teuthology.orchestra.run.smithi153.stdout:done 2024-09-10T20:14:08.230 DEBUG:teuthology.orchestra.run.smithi176:> dpkg -s linux-image-generic 2024-09-10T20:14:08.246 INFO:teuthology.orchestra.run.smithi176.stdout:Package: linux-image-generic 2024-09-10T20:14:08.246 INFO:teuthology.orchestra.run.smithi176.stdout:Status: install ok installed 2024-09-10T20:14:08.246 INFO:teuthology.orchestra.run.smithi176.stdout:Priority: optional 2024-09-10T20:14:08.246 INFO:teuthology.orchestra.run.smithi176.stdout:Section: kernel 2024-09-10T20:14:08.246 INFO:teuthology.orchestra.run.smithi176.stdout:Installed-Size: 21 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: amd64 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout:Source: linux-meta 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout:Version: 5.15.0.119.119 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout:Recommends: thermald 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout:Description: Generic Linux kernel image 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout: This package will always depend on the latest generic kernel image 2024-09-10T20:14:08.247 INFO:teuthology.orchestra.run.smithi176.stdout: available. 2024-09-10T20:14:08.247 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-10T20:14:08.248 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T20:14:08.248 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T20:14:08.248 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-10T20:14:08.248 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi176.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T20:14:08.248 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get clean 2024-09-10T20:14:08.429 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2024-09-10T20:14:08.629 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2024-09-10T20:14:08.646 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 21 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.119.119 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2024-09-10T20:14:08.647 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2024-09-10T20:14:08.648 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2024-09-10T20:14:08.648 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-10T20:14:08.648 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-10T20:14:08.648 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-10T20:14:08.648 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-10T20:14:08.648 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi153.front.sepia.ceph.com, path=None, version=distro) 2024-09-10T20:14:08.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get clean 2024-09-10T20:14:08.687 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-10T20:14:08.687 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-10T20:14:08.751 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-10T20:14:08.824 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2024-09-10T20:14:08.832 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-10T20:14:08.948 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-10T20:14:08.948 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-10T20:14:08.955 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-10T20:14:08.972 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-10T20:14:10.114 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-10T20:14:10.133 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T20:14:10.190 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-10T20:14:10.340 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-09-10T20:14:10.360 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-10T20:14:10.377 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-09-10T20:14:10.377 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-09-10T20:14:10.424 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-09-10T20:14:10.544 INFO:teuthology.orchestra.run.smithi176.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-10T20:14:10.544 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T20:14:10.544 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-10T20:14:10.544 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-10T20:14:10.545 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-10T20:14:10.545 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-10T20:14:10.545 INFO:teuthology.orchestra.run.smithi176.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-10T20:14:10.545 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T20:14:10.583 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-10T20:14:10.585 DEBUG:teuthology.orchestra.run.smithi176:> dpkg -s linux-image-generic 2024-09-10T20:14:10.600 INFO:teuthology.orchestra.run.smithi176.stdout:Package: linux-image-generic 2024-09-10T20:14:10.600 INFO:teuthology.orchestra.run.smithi176.stdout:Status: install ok installed 2024-09-10T20:14:10.600 INFO:teuthology.orchestra.run.smithi176.stdout:Priority: optional 2024-09-10T20:14:10.600 INFO:teuthology.orchestra.run.smithi176.stdout:Section: kernel 2024-09-10T20:14:10.600 INFO:teuthology.orchestra.run.smithi176.stdout:Installed-Size: 21 2024-09-10T20:14:10.600 INFO:teuthology.orchestra.run.smithi176.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T20:14:10.600 INFO:teuthology.orchestra.run.smithi176.stdout:Architecture: amd64 2024-09-10T20:14:10.601 INFO:teuthology.orchestra.run.smithi176.stdout:Source: linux-meta 2024-09-10T20:14:10.601 INFO:teuthology.orchestra.run.smithi176.stdout:Version: 5.15.0.119.119 2024-09-10T20:14:10.601 INFO:teuthology.orchestra.run.smithi176.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-10T20:14:10.601 INFO:teuthology.orchestra.run.smithi176.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T20:14:10.601 INFO:teuthology.orchestra.run.smithi176.stdout:Recommends: thermald 2024-09-10T20:14:10.601 INFO:teuthology.orchestra.run.smithi176.stdout:Description: Generic Linux kernel image 2024-09-10T20:14:10.601 INFO:teuthology.orchestra.run.smithi176.stdout: This package will always depend on the latest generic kernel image 2024-09-10T20:14:10.601 INFO:teuthology.orchestra.run.smithi176.stdout: available. 2024-09-10T20:14:10.601 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-09-10T20:14:10.633 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2024-09-10T20:14:10.634 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2024-09-10T20:14:10.645 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.5YJ5SLSqVR 2024-09-10T20:14:10.646 DEBUG:teuthology.orchestra.run.smithi176:> sudo cp /boot/grub/grub.cfg /tmp/tmp.5YJ5SLSqVR 2024-09-10T20:14:10.703 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.5YJ5SLSqVR 2024-09-10T20:14:10.823 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.5YJ5SLSqVR is 10KB 2024-09-10T20:14:10.831 INFO:teuthology.orchestra.run.smithi153.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-10T20:14:10.831 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T20:14:10.831 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-10T20:14:10.831 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-10T20:14:10.832 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-10T20:14:10.832 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-10T20:14:10.832 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-10T20:14:10.832 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T20:14:10.837 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.5YJ5SLSqVR 2024-09-10T20:14:10.842 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-10T20:14:10.874 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-10T20:14:10.876 DEBUG:teuthology.orchestra.run.smithi153:> dpkg -s linux-image-generic 2024-09-10T20:14:10.891 INFO:teuthology.orchestra.run.smithi153.stdout:Package: linux-image-generic 2024-09-10T20:14:10.892 INFO:teuthology.orchestra.run.smithi153.stdout:Status: install ok installed 2024-09-10T20:14:10.892 INFO:teuthology.orchestra.run.smithi153.stdout:Priority: optional 2024-09-10T20:14:10.892 INFO:teuthology.orchestra.run.smithi153.stdout:Section: kernel 2024-09-10T20:14:10.892 INFO:teuthology.orchestra.run.smithi153.stdout:Installed-Size: 21 2024-09-10T20:14:10.892 INFO:teuthology.orchestra.run.smithi153.stdout:Maintainer: Ubuntu Kernel Team 2024-09-10T20:14:10.892 INFO:teuthology.orchestra.run.smithi153.stdout:Architecture: amd64 2024-09-10T20:14:10.892 INFO:teuthology.orchestra.run.smithi153.stdout:Source: linux-meta 2024-09-10T20:14:10.892 INFO:teuthology.orchestra.run.smithi153.stdout:Version: 5.15.0.119.119 2024-09-10T20:14:10.893 INFO:teuthology.orchestra.run.smithi153.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-10T20:14:10.893 INFO:teuthology.orchestra.run.smithi153.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-10T20:14:10.893 INFO:teuthology.orchestra.run.smithi153.stdout:Recommends: thermald 2024-09-10T20:14:10.893 INFO:teuthology.orchestra.run.smithi153.stdout:Description: Generic Linux kernel image 2024-09-10T20:14:10.893 INFO:teuthology.orchestra.run.smithi153.stdout: This package will always depend on the latest generic kernel image 2024-09-10T20:14:10.893 INFO:teuthology.orchestra.run.smithi153.stdout: available. 2024-09-10T20:14:10.893 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-10T20:14:10.898 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:14:10.898 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-10T20:14:10.898 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-10T20:14:10.943 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.bV3F9hZaOE 2024-09-10T20:14:10.943 DEBUG:teuthology.orchestra.run.smithi153:> sudo cp /boot/grub/grub.cfg /tmp/tmp.bV3F9hZaOE 2024-09-10T20:14:10.970 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-10T20:14:10.970 DEBUG:teuthology.orchestra.run.smithi176:> sudo update-grub 2024-09-10T20:14:11.010 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.bV3F9hZaOE 2024-09-10T20:14:11.132 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.bV3F9hZaOE is 10KB 2024-09-10T20:14:11.145 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.bV3F9hZaOE 2024-09-10T20:14:11.150 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-10T20:14:11.203 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:14:11.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-10T20:14:11.203 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-10T20:14:11.270 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-10T20:14:11.270 DEBUG:teuthology.orchestra.run.smithi153:> sudo update-grub 2024-09-10T20:14:12.165 INFO:teuthology.orchestra.run.smithi176.stderr:Sourcing file `/etc/default/grub' 2024-09-10T20:14:12.166 INFO:teuthology.orchestra.run.smithi176.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T20:14:12.168 INFO:teuthology.orchestra.run.smithi176.stderr:Generating grub configuration file ... 2024-09-10T20:14:12.327 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-10T20:14:12.333 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-10T20:14:12.553 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-10T20:14:12.558 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-10T20:14:12.597 INFO:teuthology.orchestra.run.smithi176.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-10T20:14:12.602 INFO:teuthology.orchestra.run.smithi176.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-10T20:14:12.610 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub' 2024-09-10T20:14:12.611 INFO:teuthology.orchestra.run.smithi153.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-10T20:14:12.614 INFO:teuthology.orchestra.run.smithi153.stderr:Generating grub configuration file ... 2024-09-10T20:14:12.733 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-10T20:14:12.733 INFO:teuthology.orchestra.run.smithi176.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-10T20:14:12.733 INFO:teuthology.orchestra.run.smithi176.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-10T20:14:12.751 INFO:teuthology.orchestra.run.smithi176.stderr:done 2024-09-10T20:14:12.759 DEBUG:teuthology.orchestra.run.smithi176:> sudo shutdown -r now 2024-09-10T20:14:12.809 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-10T20:14:12.815 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-10T20:14:13.077 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-10T20:14:13.083 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-10T20:14:13.123 INFO:teuthology.orchestra.run.smithi153.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-10T20:14:13.128 INFO:teuthology.orchestra.run.smithi153.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-10T20:14:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-10T20:14:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-10T20:14:13.270 INFO:teuthology.orchestra.run.smithi153.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-10T20:14:13.283 INFO:teuthology.orchestra.run.smithi153.stderr:done 2024-09-10T20:14:13.304 DEBUG:teuthology.orchestra.run.smithi153:> sudo shutdown -r now 2024-09-10T20:14:42.789 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-10T20:14:42.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-10T20:14:42.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:14:43.307 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-10T20:14:43.307 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-10T20:14:43.308 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:15:17.309 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-09-10T20:15:17.821 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.153 2024-09-10T20:15:26.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-10T20:15:26.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:15:26.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-10T20:15:26.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:15:44.701 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.176 2024-09-10T20:15:56.713 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-10T20:15:56.713 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:15:57.021 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-09-10T20:15:57.643 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-10T20:15:57.644 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-119-generic"... 2024-09-10T20:15:57.644 DEBUG:teuthology.orchestra.run.smithi176:> uname -r 2024-09-10T20:15:57.696 INFO:teuthology.orchestra.run.smithi176.stdout:5.15.0-119-generic 2024-09-10T20:15:57.696 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-10T20:15:57.697 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T20:15:57.697 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-10T20:15:58.698 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-10T20:15:58.698 DEBUG:teuthology.orchestra.run.smithi176:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T20:15:58.771 INFO:teuthology.orchestra.run.smithi176.stdout:ttyS1 2024-09-10T20:15:58.806 DEBUG:teuthology.parallel:result is None 2024-09-10T20:15:58.974 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-10T20:16:00.251 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-10T20:16:00.251 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-119-generic"... 2024-09-10T20:16:00.251 DEBUG:teuthology.orchestra.run.smithi153:> uname -r 2024-09-10T20:16:00.297 INFO:teuthology.orchestra.run.smithi153.stdout:5.15.0-119-generic 2024-09-10T20:16:00.297 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-10T20:16:00.298 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-10T20:16:00.298 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-10T20:16:01.298 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-10T20:16:01.299 DEBUG:teuthology.orchestra.run.smithi153:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-10T20:16:01.433 INFO:teuthology.orchestra.run.smithi153.stdout:ttyS1 2024-09-10T20:16:01.474 DEBUG:teuthology.parallel:result is None 2024-09-10T20:16:01.475 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-10T20:16:01.484 INFO:teuthology.task.internal:Creating test directory... 2024-09-10T20:16:01.484 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T20:16:01.487 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-10T20:16:01.493 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-10T20:16:01.499 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-10T20:16:01.506 INFO:teuthology.task.internal:Creating archive directory... 2024-09-10T20:16:01.506 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T20:16:01.535 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-10T20:16:01.554 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-10T20:16:01.566 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-10T20:16:01.566 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T20:16:01.590 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-10T20:16:01.605 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:16:01.613 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:16:01.619 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:16:01.629 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-10T20:16:01.631 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-10T20:16:01.638 INFO:teuthology.task.internal:Configuring sudo... 2024-09-10T20:16:01.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T20:16:01.658 DEBUG:teuthology.orchestra.run.smithi176:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-10T20:16:01.688 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-10T20:16:01.700 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-10T20:16:01.700 DEBUG:teuthology.orchestra.run.smithi153:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T20:16:01.714 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-10T20:16:01.736 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:16:01.765 DEBUG:teuthology.orchestra.run.smithi153:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:16:01.816 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:16:01.817 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T20:16:01.880 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-10T20:16:01.886 DEBUG:teuthology.orchestra.run.smithi176:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-10T20:16:01.933 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:16:01.933 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-10T20:16:01.998 DEBUG:teuthology.orchestra.run.smithi153:> sudo service rsyslog restart 2024-09-10T20:16:02.000 DEBUG:teuthology.orchestra.run.smithi176:> sudo service rsyslog restart 2024-09-10T20:16:02.087 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-10T20:16:02.095 INFO:teuthology.task.internal:Starting timer... 2024-09-10T20:16:02.095 INFO:teuthology.run_tasks:Running task pcp... 2024-09-10T20:16:02.108 INFO:teuthology.run_tasks:Running task selinux... 2024-09-10T20:16:02.118 DEBUG:teuthology.task.selinux:Excluding smithi153: OS 'ubuntu' does not support SELinux 2024-09-10T20:16:02.118 DEBUG:teuthology.task.selinux:Excluding smithi176: OS 'ubuntu' does not support SELinux 2024-09-10T20:16:02.118 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-10T20:16:02.118 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-10T20:16:02.118 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-10T20:16:02.118 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-10T20:16:02.127 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-10T20:16:02.139 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-10T20:16:02.140 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi153.front.sepia.ceph.com,smithi176.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-10T20:23:30.111 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi153.front.sepia.ceph.com'), Remote(name='ubuntu@smithi176.front.sepia.ceph.com')] 2024-09-10T20:23:30.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-10T20:23:30.114 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi153.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:23:30.197 DEBUG:teuthology.orchestra.run.smithi153:> true 2024-09-10T20:23:30.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi153.front.sepia.ceph.com' 2024-09-10T20:23:30.273 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-10T20:23:30.273 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi176.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-10T20:23:30.352 DEBUG:teuthology.orchestra.run.smithi176:> true 2024-09-10T20:23:30.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi176.front.sepia.ceph.com' 2024-09-10T20:23:30.429 INFO:teuthology.run_tasks:Running task clock... 2024-09-10T20:23:30.446 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-10T20:23:30.447 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T20:23:30.447 DEBUG:teuthology.orchestra.run.smithi153:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:23:30.449 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-10T20:23:30.450 DEBUG:teuthology.orchestra.run.smithi176:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:23:30.476 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-10T20:23:30.477 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: Command line: ntpd -gq 2024-09-10T20:23:30.477 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: ---------------------------------------------------- 2024-09-10T20:23:30.477 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: ntp-4 is maintained by Network Time Foundation, 2024-09-10T20:23:30.477 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-10T20:23:30.477 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: corporation. Support and training for ntp-4 are 2024-09-10T20:23:30.477 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: available at https://www.nwtime.org/support 2024-09-10T20:23:30.477 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: ---------------------------------------------------- 2024-09-10T20:23:30.478 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: proto: precision = 0.061 usec (-24) 2024-09-10T20:23:30.478 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: basedate set to 2022-02-04 2024-09-10T20:23:30.478 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: gps base set to 2022-02-06 (week 2196) 2024-09-10T20:23:30.478 INFO:teuthology.orchestra.run.smithi153.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T20:23:30.478 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T20:23:30.479 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: restrict ::: KOD does nothing without LIMITED. 2024-09-10T20:23:30.479 INFO:teuthology.orchestra.run.smithi153.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-10T20:23:30.480 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: Listen and drop on 0 v6wildcard [::]:123 2024-09-10T20:23:30.480 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-10T20:23:30.480 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: Listen normally on 2 lo 127.0.0.1:123 2024-09-10T20:23:30.480 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: Listen normally on 3 enp3s0f1 172.21.15.153:123 2024-09-10T20:23:30.480 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: Listen normally on 4 lo [::1]:123 2024-09-10T20:23:30.480 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6deb%5]:123 2024-09-10T20:23:30.480 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:30 ntpd[18434]: Listening on routing socket on fd #22 for interface updates 2024-09-10T20:23:30.498 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-10T20:23:30.498 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: Command line: ntpd -gq 2024-09-10T20:23:30.498 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: ---------------------------------------------------- 2024-09-10T20:23:30.498 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: ntp-4 is maintained by Network Time Foundation, 2024-09-10T20:23:30.498 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-10T20:23:30.498 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: corporation. Support and training for ntp-4 are 2024-09-10T20:23:30.498 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: available at https://www.nwtime.org/support 2024-09-10T20:23:30.498 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: ---------------------------------------------------- 2024-09-10T20:23:30.498 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: proto: precision = 0.043 usec (-24) 2024-09-10T20:23:30.499 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: basedate set to 2022-02-04 2024-09-10T20:23:30.499 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: gps base set to 2022-02-06 (week 2196) 2024-09-10T20:23:30.499 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T20:23:30.499 INFO:teuthology.orchestra.run.smithi176.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-10T20:23:30.500 INFO:teuthology.orchestra.run.smithi176.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-10T20:23:30.500 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: restrict ::: KOD does nothing without LIMITED. 2024-09-10T20:23:30.500 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: Listen and drop on 0 v6wildcard [::]:123 2024-09-10T20:23:30.501 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-10T20:23:30.501 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: Listen normally on 2 lo 127.0.0.1:123 2024-09-10T20:23:30.501 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: Listen normally on 3 enp3s0f1 172.21.15.176:123 2024-09-10T20:23:30.501 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: Listen normally on 4 lo [::1]:123 2024-09-10T20:23:30.501 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:6f61%5]:123 2024-09-10T20:23:30.501 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:30 ntpd[18424]: Listening on routing socket on fd #22 for interface updates 2024-09-10T20:23:31.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:31 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:31.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:31 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:31.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:31 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:31.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:31 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:31.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:31 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:31.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:31 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:31.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:31 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:31.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:31 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:31.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:31.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:31 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:32.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:32 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:32.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:32 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:32.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:32 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:32.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:32 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:32.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:32 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:32.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:32 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:33 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:33 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:33 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:33 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:33.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:33 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:33.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:33 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:33.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:33 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:34.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:34.479 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:34.479 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:34 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:34.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:34 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:34.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:34 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:34.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:34 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:34.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:34 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:34.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:34 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:34.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:34 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:35.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:35 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:35.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:35 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:35.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:35 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:35.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:35 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:35.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:35 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:35.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:35 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:35.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:35.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:35 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:36.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:36 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:36.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:36 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:36.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:36 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:36.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:36 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:36.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:36 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:36.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:36 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:36.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:36 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:37.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:37 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:37.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:37 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:37.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:37 ntpd[18434]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:37.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:37 ntpd[18434]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:37.477 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:37 ntpd[18434]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:37.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:37 ntpd[18434]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:37.478 INFO:teuthology.orchestra.run.smithi153.stdout:10 Sep 20:23:37 ntpd[18434]: ntpd: time slew +0.001635 s 2024-09-10T20:23:37.478 INFO:teuthology.orchestra.run.smithi153.stdout:ntpd: time slew +0.001635s 2024-09-10T20:23:37.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:37 ntpd[18434]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-10T20:23:37.478 INFO:teuthology.orchestra.run.smithi153.stderr:10 Sep 20:23:37 ntpd[18434]: can't open /var/log/ntpstats/loopstats.20240910: Permission denied 2024-09-10T20:23:37.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:37.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:37.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:37.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:37.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-10T20:23:37.498 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: can't open /var/log/ntpstats/rawstats.20240910: Permission denied 2024-09-10T20:23:37.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-10T20:23:37.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: can't open /var/log/ntpstats/peerstats.20240910: Permission denied 2024-09-10T20:23:37.499 INFO:teuthology.orchestra.run.smithi176.stdout:10 Sep 20:23:37 ntpd[18424]: ntpd: time slew +0.003462 s 2024-09-10T20:23:37.499 INFO:teuthology.orchestra.run.smithi176.stdout:ntpd: time slew +0.003462s 2024-09-10T20:23:37.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-10T20:23:37.499 INFO:teuthology.orchestra.run.smithi176.stderr:10 Sep 20:23:37 ntpd[18424]: can't open /var/log/ntpstats/loopstats.20240910: Permission denied 2024-09-10T20:23:37.534 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T20:23:37.534 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2024-09-10T20:23:37.534 INFO:teuthology.orchestra.run.smithi153.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T20:23:37.534 INFO:teuthology.orchestra.run.smithi153.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T20:23:37.535 INFO:teuthology.orchestra.run.smithi153.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T20:23:37.535 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T20:23:37.550 INFO:teuthology.orchestra.run.smithi176.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T20:23:37.551 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================== 2024-09-10T20:23:37.551 INFO:teuthology.orchestra.run.smithi176.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T20:23:37.551 INFO:teuthology.orchestra.run.smithi176.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T20:23:37.551 INFO:teuthology.orchestra.run.smithi176.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T20:23:37.551 INFO:teuthology.orchestra.run.smithi176.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-10T20:23:37.552 INFO:teuthology.run_tasks:Running task install... 2024-09-10T20:23:37.560 DEBUG:teuthology.task.install:project ceph 2024-09-10T20:23:37.560 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}} 2024-09-10T20:23:37.560 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'} 2024-09-10T20:23:37.560 INFO:teuthology.task.install:Using flavor: default 2024-09-10T20:23:37.566 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-10T20:23:37.566 INFO:teuthology.task.install:extra packages: [] 2024-09-10T20:23:37.566 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-key list | grep Ceph 2024-09-10T20:23:37.567 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-key list | grep Ceph 2024-09-10T20:23:37.627 INFO:teuthology.orchestra.run.smithi153.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-10T20:23:37.652 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-10T20:23:37.653 INFO:teuthology.orchestra.run.smithi153.stdout:uid [ unknown] Ceph.com (release key) 2024-09-10T20:23:37.653 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-10T20:23:37.653 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:23:37.658 INFO:teuthology.orchestra.run.smithi176.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-10T20:23:37.688 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-10T20:23:37.689 INFO:teuthology.orchestra.run.smithi176.stdout:uid [ unknown] Ceph.com (release key) 2024-09-10T20:23:37.689 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-10T20:23:37.689 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:23:37.805 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default/ 2024-09-10T20:23:37.805 INFO:teuthology.task.install.deb:Package version is 19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:23:37.825 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default/ 2024-09-10T20:23:37.825 INFO:teuthology.task.install.deb:Package version is 19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:23:37.916 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:23:37.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-10T20:23:37.931 DEBUG:teuthology.orchestra.run.smithi153:> sudo apt-get update 2024-09-10T20:23:37.935 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:23:37.935 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-10T20:23:37.949 DEBUG:teuthology.orchestra.run.smithi176:> sudo apt-get update 2024-09-10T20:23:38.103 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-10T20:23:38.103 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-10T20:23:38.112 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-10T20:23:38.117 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-10T20:23:38.127 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-10T20:23:38.129 INFO:teuthology.orchestra.run.smithi153.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-10T20:23:38.144 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-10T20:23:38.156 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy InRelease 2024-09-10T20:23:38.171 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy InRelease 2024-09-10T20:23:38.177 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-10T20:23:38.192 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-10T20:23:38.199 INFO:teuthology.orchestra.run.smithi153.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-10T20:23:38.213 INFO:teuthology.orchestra.run.smithi176.stdout:Ign:6 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-10T20:23:38.220 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2024-09-10T20:23:38.235 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-10T20:23:38.240 INFO:teuthology.orchestra.run.smithi176.stdout:Hit:8 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-10T20:23:38.246 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.2 kB] 2024-09-10T20:23:38.261 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,407 B] 2024-09-10T20:23:38.449 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 29.3 kB in 0s (91.5 kB/s) 2024-09-10T20:23:38.455 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 29.3 kB in 0s (86.1 kB/s) 2024-09-10T20:23:39.439 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-10T20:23:39.456 DEBUG:teuthology.orchestra.run.smithi176:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4858-gbad1f558-1jammy cephadm=19.3.0-4858-gbad1f558-1jammy ceph-mds=19.3.0-4858-gbad1f558-1jammy ceph-mgr=19.3.0-4858-gbad1f558-1jammy ceph-common=19.3.0-4858-gbad1f558-1jammy ceph-fuse=19.3.0-4858-gbad1f558-1jammy ceph-test=19.3.0-4858-gbad1f558-1jammy ceph-volume=19.3.0-4858-gbad1f558-1jammy radosgw=19.3.0-4858-gbad1f558-1jammy python3-rados=19.3.0-4858-gbad1f558-1jammy python3-rgw=19.3.0-4858-gbad1f558-1jammy python3-cephfs=19.3.0-4858-gbad1f558-1jammy python3-rbd=19.3.0-4858-gbad1f558-1jammy libcephfs2=19.3.0-4858-gbad1f558-1jammy libcephfs-dev=19.3.0-4858-gbad1f558-1jammy librados2=19.3.0-4858-gbad1f558-1jammy librbd1=19.3.0-4858-gbad1f558-1jammy rbd-fuse=19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:23:39.493 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-09-10T20:23:39.510 DEBUG:teuthology.orchestra.run.smithi153:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4858-gbad1f558-1jammy cephadm=19.3.0-4858-gbad1f558-1jammy ceph-mds=19.3.0-4858-gbad1f558-1jammy ceph-mgr=19.3.0-4858-gbad1f558-1jammy ceph-common=19.3.0-4858-gbad1f558-1jammy ceph-fuse=19.3.0-4858-gbad1f558-1jammy ceph-test=19.3.0-4858-gbad1f558-1jammy ceph-volume=19.3.0-4858-gbad1f558-1jammy radosgw=19.3.0-4858-gbad1f558-1jammy python3-rados=19.3.0-4858-gbad1f558-1jammy python3-rgw=19.3.0-4858-gbad1f558-1jammy python3-cephfs=19.3.0-4858-gbad1f558-1jammy python3-rbd=19.3.0-4858-gbad1f558-1jammy libcephfs2=19.3.0-4858-gbad1f558-1jammy libcephfs-dev=19.3.0-4858-gbad1f558-1jammy librados2=19.3.0-4858-gbad1f558-1jammy librbd1=19.3.0-4858-gbad1f558-1jammy rbd-fuse=19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:23:39.519 INFO:teuthology.orchestra.run.smithi176.stdout:Reading package lists... 2024-09-10T20:23:39.572 INFO:teuthology.orchestra.run.smithi153.stdout:Reading package lists... 2024-09-10T20:23:39.728 INFO:teuthology.orchestra.run.smithi176.stdout:Building dependency tree... 2024-09-10T20:23:39.729 INFO:teuthology.orchestra.run.smithi176.stdout:Reading state information... 2024-09-10T20:23:39.781 INFO:teuthology.orchestra.run.smithi153.stdout:Building dependency tree... 2024-09-10T20:23:39.782 INFO:teuthology.orchestra.run.smithi153.stdout:Reading state information... 2024-09-10T20:23:39.924 INFO:teuthology.orchestra.run.smithi176.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T20:23:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-10T20:23:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-10T20:23:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-10T20:23:39.924 INFO:teuthology.orchestra.run.smithi176.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-10T20:23:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-10T20:23:39.925 INFO:teuthology.orchestra.run.smithi176.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T20:23:39.925 INFO:teuthology.orchestra.run.smithi176.stdout:The following additional packages will be installed: 2024-09-10T20:23:39.925 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-10T20:23:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-10T20:23:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-10T20:23:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-10T20:23:39.926 INFO:teuthology.orchestra.run.smithi176.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-10T20:23:39.927 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: xmlstarlet zip 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout:Suggested packages: 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-10T20:23:39.928 INFO:teuthology.orchestra.run.smithi176.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-10T20:23:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-10T20:23:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-10T20:23:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-10T20:23:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-10T20:23:39.929 INFO:teuthology.orchestra.run.smithi176.stdout:Recommended packages: 2024-09-10T20:23:39.929 INFO:teuthology.orchestra.run.smithi176.stdout: btrfs-tools 2024-09-10T20:23:39.976 INFO:teuthology.orchestra.run.smithi153.stdout:The following packages were automatically installed and are no longer required: 2024-09-10T20:23:39.976 INFO:teuthology.orchestra.run.smithi153.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-10T20:23:39.976 INFO:teuthology.orchestra.run.smithi153.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-10T20:23:39.976 INFO:teuthology.orchestra.run.smithi153.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-10T20:23:39.977 INFO:teuthology.orchestra.run.smithi153.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-10T20:23:39.977 INFO:teuthology.orchestra.run.smithi153.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-10T20:23:39.977 INFO:teuthology.orchestra.run.smithi153.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-10T20:23:39.977 INFO:teuthology.orchestra.run.smithi153.stdout:The following additional packages will be installed: 2024-09-10T20:23:39.978 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-10T20:23:39.978 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-10T20:23:39.978 INFO:teuthology.orchestra.run.smithi153.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-10T20:23:39.978 INFO:teuthology.orchestra.run.smithi153.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-10T20:23:39.978 INFO:teuthology.orchestra.run.smithi153.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-10T20:23:39.979 INFO:teuthology.orchestra.run.smithi153.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-10T20:23:39.979 INFO:teuthology.orchestra.run.smithi153.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-10T20:23:39.979 INFO:teuthology.orchestra.run.smithi153.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-10T20:23:39.979 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-10T20:23:39.979 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-10T20:23:39.979 INFO:teuthology.orchestra.run.smithi153.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-10T20:23:39.979 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-10T20:23:39.979 INFO:teuthology.orchestra.run.smithi153.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-10T20:23:39.979 INFO:teuthology.orchestra.run.smithi153.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-10T20:23:39.980 INFO:teuthology.orchestra.run.smithi153.stdout: xmlstarlet zip 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout:Suggested packages: 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout:Recommended packages: 2024-09-10T20:23:39.981 INFO:teuthology.orchestra.run.smithi153.stdout: btrfs-tools 2024-09-10T20:23:40.031 INFO:teuthology.orchestra.run.smithi176.stdout:The following NEW packages will be installed: 2024-09-10T20:23:40.031 INFO:teuthology.orchestra.run.smithi176.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-10T20:23:40.031 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-10T20:23:40.031 INFO:teuthology.orchestra.run.smithi176.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-10T20:23:40.031 INFO:teuthology.orchestra.run.smithi176.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-10T20:23:40.031 INFO:teuthology.orchestra.run.smithi176.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-10T20:23:40.031 INFO:teuthology.orchestra.run.smithi176.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-10T20:23:40.033 INFO:teuthology.orchestra.run.smithi176.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-10T20:23:40.033 INFO:teuthology.orchestra.run.smithi176.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-10T20:23:40.033 INFO:teuthology.orchestra.run.smithi176.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-10T20:23:40.033 INFO:teuthology.orchestra.run.smithi176.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-10T20:23:40.033 INFO:teuthology.orchestra.run.smithi176.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-10T20:23:40.033 INFO:teuthology.orchestra.run.smithi176.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-10T20:23:40.033 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-10T20:23:40.033 INFO:teuthology.orchestra.run.smithi176.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-10T20:23:40.034 INFO:teuthology.orchestra.run.smithi176.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-10T20:23:40.084 INFO:teuthology.orchestra.run.smithi153.stdout:The following NEW packages will be installed: 2024-09-10T20:23:40.084 INFO:teuthology.orchestra.run.smithi153.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-10T20:23:40.084 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-10T20:23:40.084 INFO:teuthology.orchestra.run.smithi153.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-10T20:23:40.084 INFO:teuthology.orchestra.run.smithi153.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-10T20:23:40.084 INFO:teuthology.orchestra.run.smithi153.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-10T20:23:40.084 INFO:teuthology.orchestra.run.smithi153.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-10T20:23:40.086 INFO:teuthology.orchestra.run.smithi153.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-10T20:23:40.086 INFO:teuthology.orchestra.run.smithi153.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-10T20:23:40.086 INFO:teuthology.orchestra.run.smithi153.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-10T20:23:40.086 INFO:teuthology.orchestra.run.smithi153.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-10T20:23:40.086 INFO:teuthology.orchestra.run.smithi153.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-10T20:23:40.086 INFO:teuthology.orchestra.run.smithi153.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-10T20:23:40.086 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-10T20:23:40.086 INFO:teuthology.orchestra.run.smithi153.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-10T20:23:40.086 INFO:teuthology.orchestra.run.smithi153.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-10T20:23:40.087 INFO:teuthology.orchestra.run.smithi153.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-10T20:23:40.093 INFO:teuthology.orchestra.run.smithi176.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-10T20:23:40.093 INFO:teuthology.orchestra.run.smithi176.stdout:Need to get 251 MB of archives. 2024-09-10T20:23:40.093 INFO:teuthology.orchestra.run.smithi176.stdout:After this operation, 932 MB of additional disk space will be used. 2024-09-10T20:23:40.093 INFO:teuthology.orchestra.run.smithi176.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-10T20:23:40.145 INFO:teuthology.orchestra.run.smithi153.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-10T20:23:40.145 INFO:teuthology.orchestra.run.smithi153.stdout:Need to get 251 MB of archives. 2024-09-10T20:23:40.145 INFO:teuthology.orchestra.run.smithi153.stdout:After this operation, 932 MB of additional disk space will be used. 2024-09-10T20:23:40.145 INFO:teuthology.orchestra.run.smithi153.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-10T20:23:40.176 INFO:teuthology.orchestra.run.smithi176.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4858-gbad1f558-1jammy [3,634 kB] 2024-09-10T20:23:40.180 INFO:teuthology.orchestra.run.smithi176.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-10T20:23:40.183 INFO:teuthology.orchestra.run.smithi176.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-10T20:23:40.200 INFO:teuthology.orchestra.run.smithi176.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-10T20:23:40.227 INFO:teuthology.orchestra.run.smithi153.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4858-gbad1f558-1jammy [3,634 kB] 2024-09-10T20:23:40.234 INFO:teuthology.orchestra.run.smithi153.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-10T20:23:40.236 INFO:teuthology.orchestra.run.smithi153.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-10T20:23:40.254 INFO:teuthology.orchestra.run.smithi153.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-10T20:23:40.261 INFO:teuthology.orchestra.run.smithi176.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-10T20:23:40.267 INFO:teuthology.orchestra.run.smithi176.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-10T20:23:40.278 INFO:teuthology.orchestra.run.smithi176.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-10T20:23:40.283 INFO:teuthology.orchestra.run.smithi176.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-10T20:23:40.284 INFO:teuthology.orchestra.run.smithi176.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-10T20:23:40.285 INFO:teuthology.orchestra.run.smithi176.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-10T20:23:40.285 INFO:teuthology.orchestra.run.smithi176.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-10T20:23:40.293 INFO:teuthology.orchestra.run.smithi176.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-10T20:23:40.295 INFO:teuthology.orchestra.run.smithi176.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-10T20:23:40.297 INFO:teuthology.orchestra.run.smithi176.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-10T20:23:40.300 INFO:teuthology.orchestra.run.smithi176.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-10T20:23:40.301 INFO:teuthology.orchestra.run.smithi176.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-10T20:23:40.313 INFO:teuthology.orchestra.run.smithi153.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-10T20:23:40.317 INFO:teuthology.orchestra.run.smithi176.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-10T20:23:40.318 INFO:teuthology.orchestra.run.smithi153.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-10T20:23:40.320 INFO:teuthology.orchestra.run.smithi176.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-10T20:23:40.322 INFO:teuthology.orchestra.run.smithi176.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-10T20:23:40.322 INFO:teuthology.orchestra.run.smithi176.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-10T20:23:40.322 INFO:teuthology.orchestra.run.smithi176.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-10T20:23:40.324 INFO:teuthology.orchestra.run.smithi176.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-10T20:23:40.324 INFO:teuthology.orchestra.run.smithi176.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-10T20:23:40.324 INFO:teuthology.orchestra.run.smithi176.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-10T20:23:40.330 INFO:teuthology.orchestra.run.smithi153.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-10T20:23:40.334 INFO:teuthology.orchestra.run.smithi176.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-10T20:23:40.335 INFO:teuthology.orchestra.run.smithi153.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-10T20:23:40.335 INFO:teuthology.orchestra.run.smithi176.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-10T20:23:40.336 INFO:teuthology.orchestra.run.smithi153.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-10T20:23:40.337 INFO:teuthology.orchestra.run.smithi153.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-10T20:23:40.337 INFO:teuthology.orchestra.run.smithi153.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-10T20:23:40.346 INFO:teuthology.orchestra.run.smithi153.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-10T20:23:40.348 INFO:teuthology.orchestra.run.smithi153.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-10T20:23:40.350 INFO:teuthology.orchestra.run.smithi153.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-10T20:23:40.351 INFO:teuthology.orchestra.run.smithi176.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-10T20:23:40.351 INFO:teuthology.orchestra.run.smithi153.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-10T20:23:40.352 INFO:teuthology.orchestra.run.smithi176.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-10T20:23:40.352 INFO:teuthology.orchestra.run.smithi153.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-10T20:23:40.354 INFO:teuthology.orchestra.run.smithi176.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-10T20:23:40.355 INFO:teuthology.orchestra.run.smithi176.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-10T20:23:40.355 INFO:teuthology.orchestra.run.smithi176.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-10T20:23:40.356 INFO:teuthology.orchestra.run.smithi176.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-10T20:23:40.357 INFO:teuthology.orchestra.run.smithi176.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-10T20:23:40.357 INFO:teuthology.orchestra.run.smithi176.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-10T20:23:40.368 INFO:teuthology.orchestra.run.smithi176.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-10T20:23:40.369 INFO:teuthology.orchestra.run.smithi153.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-10T20:23:40.370 INFO:teuthology.orchestra.run.smithi176.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-10T20:23:40.371 INFO:teuthology.orchestra.run.smithi153.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-10T20:23:40.373 INFO:teuthology.orchestra.run.smithi153.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-10T20:23:40.374 INFO:teuthology.orchestra.run.smithi153.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-10T20:23:40.374 INFO:teuthology.orchestra.run.smithi153.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-10T20:23:40.375 INFO:teuthology.orchestra.run.smithi153.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-10T20:23:40.375 INFO:teuthology.orchestra.run.smithi153.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-10T20:23:40.378 INFO:teuthology.orchestra.run.smithi176.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4858-gbad1f558-1jammy [3,330 kB] 2024-09-10T20:23:40.385 INFO:teuthology.orchestra.run.smithi176.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-10T20:23:40.386 INFO:teuthology.orchestra.run.smithi176.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-10T20:23:40.386 INFO:teuthology.orchestra.run.smithi153.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-10T20:23:40.387 INFO:teuthology.orchestra.run.smithi153.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-10T20:23:40.387 INFO:teuthology.orchestra.run.smithi153.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-10T20:23:40.391 INFO:teuthology.orchestra.run.smithi176.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-10T20:23:40.391 INFO:teuthology.orchestra.run.smithi176.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-10T20:23:40.392 INFO:teuthology.orchestra.run.smithi176.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-10T20:23:40.392 INFO:teuthology.orchestra.run.smithi176.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-10T20:23:40.393 INFO:teuthology.orchestra.run.smithi176.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-10T20:23:40.395 INFO:teuthology.orchestra.run.smithi176.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-10T20:23:40.402 INFO:teuthology.orchestra.run.smithi176.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-10T20:23:40.403 INFO:teuthology.orchestra.run.smithi153.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-10T20:23:40.404 INFO:teuthology.orchestra.run.smithi176.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-10T20:23:40.404 INFO:teuthology.orchestra.run.smithi153.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-10T20:23:40.407 INFO:teuthology.orchestra.run.smithi153.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-10T20:23:40.407 INFO:teuthology.orchestra.run.smithi153.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-10T20:23:40.407 INFO:teuthology.orchestra.run.smithi153.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-10T20:23:40.408 INFO:teuthology.orchestra.run.smithi153.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-10T20:23:40.409 INFO:teuthology.orchestra.run.smithi153.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-10T20:23:40.413 INFO:teuthology.orchestra.run.smithi176.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-10T20:23:40.416 INFO:teuthology.orchestra.run.smithi176.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-10T20:23:40.416 INFO:teuthology.orchestra.run.smithi176.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-10T20:23:40.419 INFO:teuthology.orchestra.run.smithi176.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-10T20:23:40.420 INFO:teuthology.orchestra.run.smithi176.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-10T20:23:40.421 INFO:teuthology.orchestra.run.smithi153.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-10T20:23:40.421 INFO:teuthology.orchestra.run.smithi153.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-10T20:23:40.422 INFO:teuthology.orchestra.run.smithi153.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-10T20:23:40.433 INFO:teuthology.orchestra.run.smithi153.stdout:Get:38 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4858-gbad1f558-1jammy [3,330 kB] 2024-09-10T20:23:40.438 INFO:teuthology.orchestra.run.smithi153.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-10T20:23:40.438 INFO:teuthology.orchestra.run.smithi153.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-10T20:23:40.444 INFO:teuthology.orchestra.run.smithi153.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-10T20:23:40.444 INFO:teuthology.orchestra.run.smithi153.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-10T20:23:40.444 INFO:teuthology.orchestra.run.smithi153.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-10T20:23:40.445 INFO:teuthology.orchestra.run.smithi153.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-10T20:23:40.446 INFO:teuthology.orchestra.run.smithi153.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-10T20:23:40.455 INFO:teuthology.orchestra.run.smithi153.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-10T20:23:40.456 INFO:teuthology.orchestra.run.smithi153.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-10T20:23:40.457 INFO:teuthology.orchestra.run.smithi153.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-10T20:23:40.460 INFO:teuthology.orchestra.run.smithi176.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4858-gbad1f558-1jammy [722 kB] 2024-09-10T20:23:40.472 INFO:teuthology.orchestra.run.smithi153.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-10T20:23:40.473 INFO:teuthology.orchestra.run.smithi176.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-10T20:23:40.474 INFO:teuthology.orchestra.run.smithi176.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-10T20:23:40.475 INFO:teuthology.orchestra.run.smithi153.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-10T20:23:40.475 INFO:teuthology.orchestra.run.smithi176.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-10T20:23:40.475 INFO:teuthology.orchestra.run.smithi176.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-10T20:23:40.476 INFO:teuthology.orchestra.run.smithi153.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-10T20:23:40.476 INFO:teuthology.orchestra.run.smithi153.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-10T20:23:40.476 INFO:teuthology.orchestra.run.smithi176.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-10T20:23:40.478 INFO:teuthology.orchestra.run.smithi153.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-10T20:23:40.486 INFO:teuthology.orchestra.run.smithi176.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4858-gbad1f558-1jammy [357 kB] 2024-09-10T20:23:40.494 INFO:teuthology.orchestra.run.smithi176.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-10T20:23:40.494 INFO:teuthology.orchestra.run.smithi176.stdout:Get:62 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4858-gbad1f558-1jammy [32.8 kB] 2024-09-10T20:23:40.495 INFO:teuthology.orchestra.run.smithi176.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4858-gbad1f558-1jammy [180 kB] 2024-09-10T20:23:40.496 INFO:teuthology.orchestra.run.smithi176.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-10T20:23:40.496 INFO:teuthology.orchestra.run.smithi176.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-10T20:23:40.498 INFO:teuthology.orchestra.run.smithi176.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4858-gbad1f558-1jammy [71.2 kB] 2024-09-10T20:23:40.508 INFO:teuthology.orchestra.run.smithi176.stdout:Get:67 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4858-gbad1f558-1jammy [338 kB] 2024-09-10T20:23:40.511 INFO:teuthology.orchestra.run.smithi176.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-10T20:23:40.511 INFO:teuthology.orchestra.run.smithi176.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-10T20:23:40.511 INFO:teuthology.orchestra.run.smithi176.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-10T20:23:40.512 INFO:teuthology.orchestra.run.smithi176.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-10T20:23:40.512 INFO:teuthology.orchestra.run.smithi176.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-10T20:23:40.513 INFO:teuthology.orchestra.run.smithi176.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-10T20:23:40.518 INFO:teuthology.orchestra.run.smithi176.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4858-gbad1f558-1jammy [7,987 kB] 2024-09-10T20:23:40.526 INFO:teuthology.orchestra.run.smithi176.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-10T20:23:40.527 INFO:teuthology.orchestra.run.smithi176.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-10T20:23:40.530 INFO:teuthology.orchestra.run.smithi153.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-10T20:23:40.530 INFO:teuthology.orchestra.run.smithi176.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-10T20:23:40.530 INFO:teuthology.orchestra.run.smithi153.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-10T20:23:40.532 INFO:teuthology.orchestra.run.smithi153.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-10T20:23:40.532 INFO:teuthology.orchestra.run.smithi153.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-10T20:23:40.533 INFO:teuthology.orchestra.run.smithi153.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-10T20:23:40.544 INFO:teuthology.orchestra.run.smithi176.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-10T20:23:40.545 INFO:teuthology.orchestra.run.smithi176.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-10T20:23:40.546 INFO:teuthology.orchestra.run.smithi176.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-10T20:23:40.549 INFO:teuthology.orchestra.run.smithi176.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-10T20:23:40.551 INFO:teuthology.orchestra.run.smithi176.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-10T20:23:40.552 INFO:teuthology.orchestra.run.smithi176.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-10T20:23:40.552 INFO:teuthology.orchestra.run.smithi176.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-10T20:23:40.554 INFO:teuthology.orchestra.run.smithi153.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-10T20:23:40.554 INFO:teuthology.orchestra.run.smithi153.stdout:Get:60 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4858-gbad1f558-1jammy [722 kB] 2024-09-10T20:23:40.555 INFO:teuthology.orchestra.run.smithi153.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-10T20:23:40.556 INFO:teuthology.orchestra.run.smithi153.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-10T20:23:40.561 INFO:teuthology.orchestra.run.smithi176.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-10T20:23:40.562 INFO:teuthology.orchestra.run.smithi176.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-10T20:23:40.565 INFO:teuthology.orchestra.run.smithi176.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-10T20:23:40.570 INFO:teuthology.orchestra.run.smithi153.stdout:Get:63 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4858-gbad1f558-1jammy [357 kB] 2024-09-10T20:23:40.570 INFO:teuthology.orchestra.run.smithi153.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-10T20:23:40.571 INFO:teuthology.orchestra.run.smithi153.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-10T20:23:40.571 INFO:teuthology.orchestra.run.smithi153.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-10T20:23:40.571 INFO:teuthology.orchestra.run.smithi153.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-10T20:23:40.572 INFO:teuthology.orchestra.run.smithi153.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-10T20:23:40.572 INFO:teuthology.orchestra.run.smithi153.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-10T20:23:40.577 INFO:teuthology.orchestra.run.smithi153.stdout:Get:70 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4858-gbad1f558-1jammy [32.8 kB] 2024-09-10T20:23:40.577 INFO:teuthology.orchestra.run.smithi153.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4858-gbad1f558-1jammy [180 kB] 2024-09-10T20:23:40.578 INFO:teuthology.orchestra.run.smithi176.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-10T20:23:40.579 INFO:teuthology.orchestra.run.smithi176.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-10T20:23:40.579 INFO:teuthology.orchestra.run.smithi176.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-10T20:23:40.581 INFO:teuthology.orchestra.run.smithi176.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-10T20:23:40.581 INFO:teuthology.orchestra.run.smithi176.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-10T20:23:40.584 INFO:teuthology.orchestra.run.smithi153.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-10T20:23:40.584 INFO:teuthology.orchestra.run.smithi153.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-10T20:23:40.587 INFO:teuthology.orchestra.run.smithi176.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-10T20:23:40.588 INFO:teuthology.orchestra.run.smithi176.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-10T20:23:40.588 INFO:teuthology.orchestra.run.smithi153.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-10T20:23:40.588 INFO:teuthology.orchestra.run.smithi153.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4858-gbad1f558-1jammy [71.2 kB] 2024-09-10T20:23:40.590 INFO:teuthology.orchestra.run.smithi153.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4858-gbad1f558-1jammy [338 kB] 2024-09-10T20:23:40.595 INFO:teuthology.orchestra.run.smithi176.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-10T20:23:40.597 INFO:teuthology.orchestra.run.smithi176.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-10T20:23:40.598 INFO:teuthology.orchestra.run.smithi176.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-10T20:23:40.601 INFO:teuthology.orchestra.run.smithi153.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-10T20:23:40.602 INFO:teuthology.orchestra.run.smithi153.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-10T20:23:40.603 INFO:teuthology.orchestra.run.smithi153.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-10T20:23:40.603 INFO:teuthology.orchestra.run.smithi153.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4858-gbad1f558-1jammy [7,987 kB] 2024-09-10T20:23:40.606 INFO:teuthology.orchestra.run.smithi153.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-10T20:23:40.608 INFO:teuthology.orchestra.run.smithi153.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-10T20:23:40.609 INFO:teuthology.orchestra.run.smithi153.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-10T20:23:40.609 INFO:teuthology.orchestra.run.smithi153.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-10T20:23:40.619 INFO:teuthology.orchestra.run.smithi153.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-10T20:23:40.619 INFO:teuthology.orchestra.run.smithi153.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-10T20:23:40.622 INFO:teuthology.orchestra.run.smithi153.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-10T20:23:40.636 INFO:teuthology.orchestra.run.smithi153.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-10T20:23:40.637 INFO:teuthology.orchestra.run.smithi153.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-10T20:23:40.637 INFO:teuthology.orchestra.run.smithi153.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-10T20:23:40.638 INFO:teuthology.orchestra.run.smithi153.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-10T20:23:40.639 INFO:teuthology.orchestra.run.smithi153.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-10T20:23:40.645 INFO:teuthology.orchestra.run.smithi153.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-10T20:23:40.645 INFO:teuthology.orchestra.run.smithi153.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-10T20:23:40.653 INFO:teuthology.orchestra.run.smithi153.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-10T20:23:40.655 INFO:teuthology.orchestra.run.smithi153.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-10T20:23:40.655 INFO:teuthology.orchestra.run.smithi153.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-10T20:23:40.767 INFO:teuthology.orchestra.run.smithi176.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4858-gbad1f558-1jammy [112 kB] 2024-09-10T20:23:40.769 INFO:teuthology.orchestra.run.smithi176.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4858-gbad1f558-1jammy [473 kB] 2024-09-10T20:23:40.779 INFO:teuthology.orchestra.run.smithi176.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4858-gbad1f558-1jammy [28.3 MB] 2024-09-10T20:23:40.886 INFO:teuthology.orchestra.run.smithi153.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4858-gbad1f558-1jammy [112 kB] 2024-09-10T20:23:40.888 INFO:teuthology.orchestra.run.smithi153.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4858-gbad1f558-1jammy [473 kB] 2024-09-10T20:23:40.898 INFO:teuthology.orchestra.run.smithi153.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4858-gbad1f558-1jammy [28.3 MB] 2024-09-10T20:23:41.759 INFO:teuthology.orchestra.run.smithi176.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4858-gbad1f558-1jammy [5,206 kB] 2024-09-10T20:23:41.837 INFO:teuthology.orchestra.run.smithi176.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4858-gbad1f558-1jammy [251 kB] 2024-09-10T20:23:41.840 INFO:teuthology.orchestra.run.smithi176.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4858-gbad1f558-1jammy [124 kB] 2024-09-10T20:23:41.849 INFO:teuthology.orchestra.run.smithi176.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4858-gbad1f558-1jammy [1,560 kB] 2024-09-10T20:23:41.894 INFO:teuthology.orchestra.run.smithi176.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4858-gbad1f558-1jammy [6,309 kB] 2024-09-10T20:23:41.984 INFO:teuthology.orchestra.run.smithi153.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4858-gbad1f558-1jammy [5,206 kB] 2024-09-10T20:23:42.049 INFO:teuthology.orchestra.run.smithi176.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4858-gbad1f558-1jammy [23.1 MB] 2024-09-10T20:23:42.104 INFO:teuthology.orchestra.run.smithi153.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4858-gbad1f558-1jammy [251 kB] 2024-09-10T20:23:42.109 INFO:teuthology.orchestra.run.smithi153.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4858-gbad1f558-1jammy [124 kB] 2024-09-10T20:23:42.112 INFO:teuthology.orchestra.run.smithi153.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4858-gbad1f558-1jammy [1,560 kB] 2024-09-10T20:23:42.180 INFO:teuthology.orchestra.run.smithi153.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4858-gbad1f558-1jammy [6,309 kB] 2024-09-10T20:23:42.405 INFO:teuthology.orchestra.run.smithi153.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4858-gbad1f558-1jammy [23.1 MB] 2024-09-10T20:23:42.822 INFO:teuthology.orchestra.run.smithi176.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4858-gbad1f558-1jammy [14.0 kB] 2024-09-10T20:23:42.823 INFO:teuthology.orchestra.run.smithi176.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4858-gbad1f558-1jammy [880 kB] 2024-09-10T20:23:42.839 INFO:teuthology.orchestra.run.smithi176.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4858-gbad1f558-1jammy [2,453 kB] 2024-09-10T20:23:42.904 INFO:teuthology.orchestra.run.smithi176.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4858-gbad1f558-1jammy [813 kB] 2024-09-10T20:23:42.916 INFO:teuthology.orchestra.run.smithi176.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4858-gbad1f558-1jammy [160 kB] 2024-09-10T20:23:42.919 INFO:teuthology.orchestra.run.smithi176.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4858-gbad1f558-1jammy [61.4 MB] 2024-09-10T20:23:43.252 INFO:teuthology.orchestra.run.smithi153.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4858-gbad1f558-1jammy [14.0 kB] 2024-09-10T20:23:43.252 INFO:teuthology.orchestra.run.smithi153.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4858-gbad1f558-1jammy [880 kB] 2024-09-10T20:23:43.270 INFO:teuthology.orchestra.run.smithi153.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4858-gbad1f558-1jammy [2,453 kB] 2024-09-10T20:23:43.311 INFO:teuthology.orchestra.run.smithi153.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4858-gbad1f558-1jammy [813 kB] 2024-09-10T20:23:43.321 INFO:teuthology.orchestra.run.smithi153.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4858-gbad1f558-1jammy [160 kB] 2024-09-10T20:23:43.323 INFO:teuthology.orchestra.run.smithi153.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4858-gbad1f558-1jammy [61.4 MB] 2024-09-10T20:23:45.368 INFO:teuthology.orchestra.run.smithi153.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4858-gbad1f558-1jammy [8,625 kB] 2024-09-10T20:23:45.439 INFO:teuthology.orchestra.run.smithi176.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4858-gbad1f558-1jammy [8,625 kB] 2024-09-10T20:23:45.480 INFO:teuthology.orchestra.run.smithi153.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4858-gbad1f558-1jammy [14.2 kB] 2024-09-10T20:23:45.480 INFO:teuthology.orchestra.run.smithi153.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4858-gbad1f558-1jammy [55.3 MB] 2024-09-10T20:23:45.653 INFO:teuthology.orchestra.run.smithi176.stdout:Get:114 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4858-gbad1f558-1jammy [14.2 kB] 2024-09-10T20:23:45.653 INFO:teuthology.orchestra.run.smithi176.stdout:Get:115 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4858-gbad1f558-1jammy [55.3 MB] 2024-09-10T20:23:47.262 INFO:teuthology.orchestra.run.smithi153.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4858-gbad1f558-1jammy [130 kB] 2024-09-10T20:23:47.265 INFO:teuthology.orchestra.run.smithi153.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4858-gbad1f558-1jammy [39.0 kB] 2024-09-10T20:23:47.266 INFO:teuthology.orchestra.run.smithi153.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4858-gbad1f558-1jammy [15.9 MB] 2024-09-10T20:23:47.696 INFO:teuthology.orchestra.run.smithi153.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4858-gbad1f558-1jammy [91.2 kB] 2024-09-10T20:23:47.847 INFO:teuthology.orchestra.run.smithi176.stdout:Get:116 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4858-gbad1f558-1jammy [130 kB] 2024-09-10T20:23:47.850 INFO:teuthology.orchestra.run.smithi176.stdout:Get:117 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4858-gbad1f558-1jammy [39.0 kB] 2024-09-10T20:23:47.851 INFO:teuthology.orchestra.run.smithi176.stdout:Get:118 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4858-gbad1f558-1jammy [15.9 MB] 2024-09-10T20:23:48.148 INFO:teuthology.orchestra.run.smithi176.stdout:Get:119 https://4.chacra.ceph.com/r/ceph/wip-bharath7-testing-2024-09-10-1409/bad1f558c327094897e4a32e99dd5cb4ae6ccd46/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4858-gbad1f558-1jammy [91.2 kB] 2024-09-10T20:23:48.281 INFO:teuthology.orchestra.run.smithi153.stdout:Fetched 251 MB in 8s (33.0 MB/s) 2024-09-10T20:23:48.453 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-10T20:23:48.553 INFO:teuthology.orchestra.run.smithi153.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-10T20:23:48.559 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-10T20:23:48.643 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-10T20:23:48.723 INFO:teuthology.orchestra.run.smithi176.stdout:Fetched 251 MB in 8s (30.9 MB/s) 2024-09-10T20:23:48.851 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-10T20:23:48.928 INFO:teuthology.orchestra.run.smithi176.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-10T20:23:48.933 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-10T20:23:49.000 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-10T20:23:49.038 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-10T20:23:49.064 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-10T20:23:49.105 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-10T20:23:49.295 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-10T20:23:49.321 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-10T20:23:49.361 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-10T20:23:49.416 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-10T20:23:49.431 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-10T20:23:49.466 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-10T20:23:49.698 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-10T20:23:49.724 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-10T20:23:49.764 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-10T20:23:49.903 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-10T20:23:49.929 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-10T20:23:49.978 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:23:50.193 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-10T20:23:50.220 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-10T20:23:50.267 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:23:50.381 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-10T20:23:50.407 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-10T20:23:50.447 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:23:50.629 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-10T20:23:50.656 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-10T20:23:50.696 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:23:50.778 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-10T20:23:50.804 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-10T20:23:50.844 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:23:51.057 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-10T20:23:51.083 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-10T20:23:51.123 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:23:51.197 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-10T20:23:51.223 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-10T20:23:51.263 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-10T20:23:51.460 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-10T20:23:51.487 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-10T20:23:51.526 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-10T20:23:51.558 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librados2. 2024-09-10T20:23:51.584 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../007-librados2_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:51.624 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librados2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:51.838 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librados2. 2024-09-10T20:23:51.864 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../007-librados2_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:51.904 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librados2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:52.078 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libnbd0. 2024-09-10T20:23:52.104 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-10T20:23:52.144 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-10T20:23:52.340 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libnbd0. 2024-09-10T20:23:52.355 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-10T20:23:52.390 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-10T20:23:52.455 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librbd1. 2024-09-10T20:23:52.481 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../009-librbd1_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:52.521 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librbd1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:52.701 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librbd1. 2024-09-10T20:23:52.728 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../009-librbd1_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:52.768 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librbd1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:52.949 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs2. 2024-09-10T20:23:52.975 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:53.015 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:53.212 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs2. 2024-09-10T20:23:53.239 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:53.268 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rados. 2024-09-10T20:23:53.279 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:53.283 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:53.318 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rados (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:53.654 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-10T20:23:53.680 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:23:53.720 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-argparse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:53.773 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rados. 2024-09-10T20:23:53.800 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:53.840 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rados (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:53.973 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cephfs. 2024-09-10T20:23:53.999 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:54.039 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cephfs (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:54.134 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-10T20:23:54.161 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:23:54.201 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-argparse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:54.351 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-10T20:23:54.377 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:23:54.417 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:54.436 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cephfs. 2024-09-10T20:23:54.452 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:54.495 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cephfs (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:54.720 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-10T20:23:54.746 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-10T20:23:54.787 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-10T20:23:54.822 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-10T20:23:54.849 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:23:54.889 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:55.073 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-prettytable. 2024-09-10T20:23:55.099 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-10T20:23:55.139 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-10T20:23:55.225 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-10T20:23:55.252 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-10T20:23:55.292 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-10T20:23:55.400 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rbd. 2024-09-10T20:23:55.426 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:55.467 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rbd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:55.611 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-prettytable. 2024-09-10T20:23:55.637 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-10T20:23:55.678 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-10T20:23:55.820 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-10T20:23:55.846 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-10T20:23:55.886 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-10T20:23:55.913 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rbd. 2024-09-10T20:23:55.929 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:55.963 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rbd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:56.147 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-10T20:23:56.162 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-10T20:23:56.197 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-10T20:23:56.300 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-10T20:23:56.326 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-10T20:23:56.366 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-10T20:23:56.458 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-10T20:23:56.473 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-10T20:23:56.508 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-10T20:23:56.644 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-10T20:23:56.659 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-10T20:23:56.702 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-10T20:23:56.845 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua5.1. 2024-09-10T20:23:56.871 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-10T20:23:56.911 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-10T20:23:56.988 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-10T20:23:57.014 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-10T20:23:57.055 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-10T20:23:57.222 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua-any. 2024-09-10T20:23:57.248 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-10T20:23:57.289 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-10T20:23:57.366 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua5.1. 2024-09-10T20:23:57.393 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-10T20:23:57.424 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-10T20:23:57.508 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package zip. 2024-09-10T20:23:57.523 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-10T20:23:57.558 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking zip (3.0-12build2) ... 2024-09-10T20:23:57.736 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua-any. 2024-09-10T20:23:57.762 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-10T20:23:57.802 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-10T20:23:57.836 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package unzip. 2024-09-10T20:23:57.862 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-10T20:23:57.903 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-10T20:23:58.046 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package zip. 2024-09-10T20:23:58.062 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-10T20:23:58.121 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking zip (3.0-12build2) ... 2024-09-10T20:23:58.298 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package luarocks. 2024-09-10T20:23:58.324 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-10T20:23:58.365 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-10T20:23:58.450 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package unzip. 2024-09-10T20:23:58.476 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-10T20:23:58.516 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-10T20:23:59.037 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package luarocks. 2024-09-10T20:23:59.039 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package librgw2. 2024-09-10T20:23:59.063 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-10T20:23:59.065 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../026-librgw2_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:59.103 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-10T20:23:59.105 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking librgw2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:59.488 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package librgw2. 2024-09-10T20:23:59.514 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../026-librgw2_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:59.546 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking librgw2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:23:59.699 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rgw. 2024-09-10T20:23:59.715 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:23:59.749 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:00.060 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-10T20:24:00.087 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-10T20:24:00.115 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rgw. 2024-09-10T20:24:00.127 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-10T20:24:00.130 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:00.173 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:00.509 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-10T20:24:00.513 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libradosstriper1. 2024-09-10T20:24:00.535 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-10T20:24:00.539 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:00.576 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-10T20:24:00.588 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libradosstriper1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:00.887 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libradosstriper1. 2024-09-10T20:24:00.913 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:00.953 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libradosstriper1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:01.116 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-common. 2024-09-10T20:24:01.143 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:01.191 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:01.290 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-common. 2024-09-10T20:24:01.317 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:01.357 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:02.821 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-base. 2024-09-10T20:24:02.845 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-base. 2024-09-10T20:24:02.847 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:02.871 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:02.905 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-base (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:02.921 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-base (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:03.457 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-10T20:24:03.484 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-10T20:24:03.524 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-10T20:24:03.525 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-10T20:24:03.551 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-10T20:24:03.583 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-10T20:24:03.835 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cheroot. 2024-09-10T20:24:03.861 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-10T20:24:03.895 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cheroot. 2024-09-10T20:24:03.902 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-10T20:24:03.921 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-10T20:24:03.961 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-10T20:24:04.196 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-10T20:24:04.223 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-10T20:24:04.263 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-10T20:24:04.281 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-10T20:24:04.307 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-10T20:24:04.347 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-10T20:24:04.566 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-10T20:24:04.592 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-10T20:24:04.632 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-10T20:24:04.633 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-10T20:24:04.660 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-10T20:24:04.700 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-10T20:24:04.893 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-10T20:24:04.920 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-10T20:24:04.960 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-10T20:24:04.961 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-10T20:24:04.987 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-10T20:24:05.019 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-10T20:24:05.246 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-tempora. 2024-09-10T20:24:05.272 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-10T20:24:05.297 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempora. 2024-09-10T20:24:05.324 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-10T20:24:05.329 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-10T20:24:05.364 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-10T20:24:05.632 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-portend. 2024-09-10T20:24:05.658 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-10T20:24:05.675 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-portend. 2024-09-10T20:24:05.698 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-10T20:24:05.701 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-10T20:24:05.741 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-10T20:24:06.001 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-10T20:24:06.027 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-10T20:24:06.044 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-10T20:24:06.068 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-10T20:24:06.070 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-10T20:24:06.111 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-10T20:24:06.321 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-10T20:24:06.336 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-10T20:24:06.355 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-10T20:24:06.370 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-10T20:24:06.371 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-10T20:24:06.413 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-10T20:24:06.724 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-natsort. 2024-09-10T20:24:06.750 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-10T20:24:06.775 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-natsort. 2024-09-10T20:24:06.790 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-10T20:24:06.802 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-10T20:24:06.841 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-10T20:24:07.085 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-logutils. 2024-09-10T20:24:07.112 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-10T20:24:07.137 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-logutils. 2024-09-10T20:24:07.160 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-10T20:24:07.163 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-10T20:24:07.196 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-10T20:24:07.438 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-10T20:24:07.464 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-10T20:24:07.481 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-10T20:24:07.504 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-10T20:24:07.507 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-10T20:24:07.547 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-10T20:24:07.807 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-mako. 2024-09-10T20:24:07.834 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-10T20:24:07.850 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-mako. 2024-09-10T20:24:07.873 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-10T20:24:07.876 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-10T20:24:07.916 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-10T20:24:08.193 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-10T20:24:08.220 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-10T20:24:08.260 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-10T20:24:08.261 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-10T20:24:08.288 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-10T20:24:08.328 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-10T20:24:08.555 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-10T20:24:08.581 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-10T20:24:08.614 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-10T20:24:08.621 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-10T20:24:08.640 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-10T20:24:08.680 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-10T20:24:08.908 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webob. 2024-09-10T20:24:08.915 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webob. 2024-09-10T20:24:08.923 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-10T20:24:08.942 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-10T20:24:08.966 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-10T20:24:08.982 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-10T20:24:09.286 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-waitress. 2024-09-10T20:24:09.312 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-10T20:24:09.364 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-10T20:24:09.518 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-waitress. 2024-09-10T20:24:09.545 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-10T20:24:09.597 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-10T20:24:09.672 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-tempita. 2024-09-10T20:24:09.699 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-10T20:24:09.739 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-10T20:24:09.871 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-tempita. 2024-09-10T20:24:09.898 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-10T20:24:09.938 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-10T20:24:10.041 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-paste. 2024-09-10T20:24:10.068 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-10T20:24:10.108 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-10T20:24:10.257 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-paste. 2024-09-10T20:24:10.284 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-10T20:24:10.324 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-10T20:24:10.411 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-10T20:24:10.426 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-10T20:24:10.469 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-10T20:24:10.618 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-10T20:24:10.645 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-10T20:24:10.694 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-10T20:24:10.780 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-10T20:24:10.807 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-10T20:24:10.847 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-10T20:24:10.988 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-10T20:24:11.014 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-10T20:24:11.054 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-10T20:24:11.175 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-webtest. 2024-09-10T20:24:11.201 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-10T20:24:11.241 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-10T20:24:11.341 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-webtest. 2024-09-10T20:24:11.367 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-10T20:24:11.407 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-10T20:24:11.510 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pecan. 2024-09-10T20:24:11.526 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-10T20:24:11.569 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-10T20:24:11.677 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pecan. 2024-09-10T20:24:11.703 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-10T20:24:11.734 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-10T20:24:11.882 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-certifi. 2024-09-10T20:24:11.897 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-10T20:24:11.932 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-10T20:24:12.048 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-certifi. 2024-09-10T20:24:12.074 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-10T20:24:12.106 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-10T20:24:12.218 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-idna. 2024-09-10T20:24:12.244 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-10T20:24:12.284 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-10T20:24:12.441 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-idna. 2024-09-10T20:24:12.468 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-10T20:24:12.508 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-10T20:24:12.554 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-urllib3. 2024-09-10T20:24:12.580 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-10T20:24:12.612 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-10T20:24:12.828 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-urllib3. 2024-09-10T20:24:12.855 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-10T20:24:12.890 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests. 2024-09-10T20:24:12.895 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-10T20:24:12.917 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-10T20:24:12.948 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-10T20:24:13.193 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-10T20:24:13.208 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-10T20:24:13.223 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-requests. 2024-09-10T20:24:13.242 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-10T20:24:13.249 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-10T20:24:13.289 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-10T20:24:14.318 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-10T20:24:14.345 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-10T20:24:14.384 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-10T20:24:14.629 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-10T20:24:14.644 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:14.662 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-10T20:24:14.678 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:14.687 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:14.720 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:15.067 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-10T20:24:15.093 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:15.125 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-10T20:24:15.133 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:15.152 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:15.191 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:15.428 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-10T20:24:15.452 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-10T20:24:15.454 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-10T20:24:15.479 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-10T20:24:15.485 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-10T20:24:15.527 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-10T20:24:15.739 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-openssl. 2024-09-10T20:24:15.765 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-10T20:24:15.805 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-10T20:24:15.830 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-openssl. 2024-09-10T20:24:15.857 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-10T20:24:15.896 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-10T20:24:16.133 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr. 2024-09-10T20:24:16.160 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:16.199 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:16.233 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr. 2024-09-10T20:24:16.260 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:16.299 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:16.594 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mon. 2024-09-10T20:24:16.621 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:16.660 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mon. 2024-09-10T20:24:16.661 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mon (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:16.676 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:16.719 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mon (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:17.189 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-osd. 2024-09-10T20:24:17.216 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:17.247 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-osd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:17.273 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-osd. 2024-09-10T20:24:17.300 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:17.342 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-osd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:18.309 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph. 2024-09-10T20:24:18.324 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../067-ceph_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:18.367 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:18.375 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph. 2024-09-10T20:24:18.402 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../067-ceph_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:18.441 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:18.686 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-fuse. 2024-09-10T20:24:18.713 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:18.728 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-fuse. 2024-09-10T20:24:18.752 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:18.754 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:18.794 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:19.089 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mds. 2024-09-10T20:24:19.115 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:19.155 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mds (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:19.164 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mds. 2024-09-10T20:24:19.190 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:19.230 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mds (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:19.525 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package cephadm. 2024-09-10T20:24:19.552 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../070-cephadm_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:19.591 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:19.666 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package cephadm. 2024-09-10T20:24:19.693 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../070-cephadm_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:19.733 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:19.919 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-10T20:24:19.946 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-10T20:24:19.985 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-10T20:24:20.019 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-10T20:24:20.045 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-10T20:24:20.085 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-10T20:24:20.272 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-10T20:24:20.287 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-10T20:24:20.330 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-10T20:24:20.380 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-10T20:24:20.396 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-10T20:24:20.438 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-10T20:24:21.327 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-babel. 2024-09-10T20:24:21.353 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-10T20:24:21.393 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-10T20:24:21.434 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-babel. 2024-09-10T20:24:21.462 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-10T20:24:21.500 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-10T20:24:21.704 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-jinja2. 2024-09-10T20:24:21.731 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-10T20:24:21.771 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-10T20:24:21.820 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-jinja2. 2024-09-10T20:24:21.848 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-10T20:24:21.879 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-10T20:24:22.090 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-10T20:24:22.117 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:22.157 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:22.190 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-10T20:24:22.217 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:22.248 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:22.477 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-10T20:24:22.503 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-10T20:24:22.543 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-10T20:24:22.568 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-10T20:24:22.595 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-10T20:24:22.634 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-10T20:24:22.812 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-routes. 2024-09-10T20:24:22.828 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-10T20:24:22.862 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-10T20:24:22.920 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-routes. 2024-09-10T20:24:22.948 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-10T20:24:22.987 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-10T20:24:23.165 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-10T20:24:23.192 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:23.223 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:23.307 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-10T20:24:23.334 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:23.373 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:26.019 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-10T20:24:26.049 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-10T20:24:26.094 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-10T20:24:26.153 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-10T20:24:26.182 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-10T20:24:26.211 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-10T20:24:26.597 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-joblib. 2024-09-10T20:24:26.625 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-10T20:24:26.663 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-10T20:24:26.681 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-joblib. 2024-09-10T20:24:26.708 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-10T20:24:26.747 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-10T20:24:27.042 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-10T20:24:27.070 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-10T20:24:27.100 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-10T20:24:27.109 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-10T20:24:27.136 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-10T20:24:27.175 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-10T20:24:27.378 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-sklearn. 2024-09-10T20:24:27.406 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-10T20:24:27.462 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-sklearn. 2024-09-10T20:24:27.490 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-10T20:24:27.522 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-10T20:24:27.623 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-10T20:24:28.231 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-10T20:24:28.259 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:28.297 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:28.358 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-10T20:24:28.374 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:28.407 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:29.208 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-cachetools. 2024-09-10T20:24:29.236 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-10T20:24:29.274 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-10T20:24:29.285 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-cachetools. 2024-09-10T20:24:29.314 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-10T20:24:29.344 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-10T20:24:29.577 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-10T20:24:29.605 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-10T20:24:29.646 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-10T20:24:29.652 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-10T20:24:29.674 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-10T20:24:29.722 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-10T20:24:30.042 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-10T20:24:30.069 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-10T20:24:30.109 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-10T20:24:30.231 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-10T20:24:30.248 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-10T20:24:30.289 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-10T20:24:30.419 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-rsa. 2024-09-10T20:24:30.447 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-10T20:24:30.486 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-10T20:24:30.609 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-rsa. 2024-09-10T20:24:30.636 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-10T20:24:30.675 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-10T20:24:30.773 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-google-auth. 2024-09-10T20:24:30.800 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-10T20:24:30.839 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-10T20:24:30.962 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-google-auth. 2024-09-10T20:24:30.990 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-10T20:24:31.028 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-10T20:24:31.117 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-10T20:24:31.145 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-10T20:24:31.175 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-10T20:24:31.348 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-10T20:24:31.376 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-10T20:24:31.414 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-10T20:24:31.453 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-websocket. 2024-09-10T20:24:31.481 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-10T20:24:31.520 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-10T20:24:31.726 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-websocket. 2024-09-10T20:24:31.753 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-10T20:24:31.792 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-10T20:24:31.823 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-10T20:24:31.851 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-10T20:24:31.889 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-10T20:24:32.112 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-10T20:24:32.140 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-10T20:24:32.179 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-10T20:24:32.561 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-10T20:24:32.578 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:32.619 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:32.809 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-10T20:24:32.838 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:32.875 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:32.973 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-10T20:24:33.000 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-10T20:24:33.039 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-10T20:24:33.212 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-10T20:24:33.240 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-10T20:24:33.278 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-10T20:24:33.342 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-10T20:24:33.370 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-10T20:24:33.400 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-10T20:24:33.581 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-10T20:24:33.609 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-10T20:24:33.628 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package jq. 2024-09-10T20:24:33.639 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-10T20:24:33.655 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-10T20:24:33.694 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-10T20:24:33.883 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package jq. 2024-09-10T20:24:33.900 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-10T20:24:33.963 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package socat. 2024-09-10T20:24:33.966 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-10T20:24:33.991 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-10T20:24:34.030 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-10T20:24:34.219 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package socat. 2024-09-10T20:24:34.248 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-10T20:24:34.294 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-10T20:24:34.350 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package xmlstarlet. 2024-09-10T20:24:34.378 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-10T20:24:34.416 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-10T20:24:34.589 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package xmlstarlet. 2024-09-10T20:24:34.617 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-10T20:24:34.656 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-10T20:24:34.754 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-test. 2024-09-10T20:24:34.770 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:34.812 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-test (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:34.976 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-test. 2024-09-10T20:24:34.993 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:35.026 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-test (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:37.300 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package ceph-volume. 2024-09-10T20:24:37.328 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:37.366 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking ceph-volume (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:37.389 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package ceph-volume. 2024-09-10T20:24:37.417 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4858-gbad1f558-1jammy_all.deb ... 2024-09-10T20:24:37.455 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking ceph-volume (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:37.662 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-10T20:24:37.690 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:37.728 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking libcephfs-dev (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:37.876 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-10T20:24:37.876 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:37.876 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking libcephfs-dev (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:38.039 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-10T20:24:38.067 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-10T20:24:38.105 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-10T20:24:38.178 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-10T20:24:38.207 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-10T20:24:38.245 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-10T20:24:38.435 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-10T20:24:38.463 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-10T20:24:38.501 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-10T20:24:38.574 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-10T20:24:38.602 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-10T20:24:38.640 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-10T20:24:38.814 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package nvme-cli. 2024-09-10T20:24:38.841 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-10T20:24:38.880 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-10T20:24:38.936 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package nvme-cli. 2024-09-10T20:24:38.964 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-10T20:24:38.994 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-10T20:24:39.293 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package pkg-config. 2024-09-10T20:24:39.321 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-10T20:24:39.360 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-10T20:24:39.383 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package pkg-config. 2024-09-10T20:24:39.411 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-10T20:24:39.449 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-10T20:24:39.621 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-10T20:24:39.637 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-10T20:24:39.679 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-10T20:24:39.701 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-10T20:24:39.720 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-10T20:24:39.760 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-10T20:24:40.091 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-10T20:24:40.119 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-10T20:24:40.157 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-10T20:24:40.182 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-10T20:24:40.210 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-10T20:24:40.248 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-10T20:24:40.460 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pastescript. 2024-09-10T20:24:40.488 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-10T20:24:40.526 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-10T20:24:40.567 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pastescript. 2024-09-10T20:24:40.595 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-10T20:24:40.634 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-10T20:24:40.838 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pluggy. 2024-09-10T20:24:40.866 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-10T20:24:40.904 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-10T20:24:40.954 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pluggy. 2024-09-10T20:24:40.982 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-10T20:24:41.020 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-10T20:24:41.182 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-psutil. 2024-09-10T20:24:41.210 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-10T20:24:41.265 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-10T20:24:41.365 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-psutil. 2024-09-10T20:24:41.393 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-10T20:24:41.431 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-10T20:24:41.627 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-py. 2024-09-10T20:24:41.655 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-10T20:24:41.693 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-10T20:24:41.776 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-py. 2024-09-10T20:24:41.804 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-10T20:24:41.842 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-10T20:24:42.021 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pygments. 2024-09-10T20:24:42.049 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-10T20:24:42.087 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-10T20:24:42.170 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pygments. 2024-09-10T20:24:42.199 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-10T20:24:42.237 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-10T20:24:42.475 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-10T20:24:42.491 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-10T20:24:42.525 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-10T20:24:42.691 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-10T20:24:42.719 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-10T20:24:42.757 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-10T20:24:42.861 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-toml. 2024-09-10T20:24:42.889 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-10T20:24:42.927 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-10T20:24:43.010 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-toml. 2024-09-10T20:24:43.026 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-10T20:24:43.068 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-10T20:24:43.230 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-pytest. 2024-09-10T20:24:43.258 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-10T20:24:43.296 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-10T20:24:43.371 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-pytest. 2024-09-10T20:24:43.399 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-10T20:24:43.437 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-10T20:24:43.583 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package python3-simplejson. 2024-09-10T20:24:43.600 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-10T20:24:43.633 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-10T20:24:43.782 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package python3-simplejson. 2024-09-10T20:24:43.810 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-10T20:24:43.848 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-10T20:24:43.877 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-10T20:24:43.893 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-10T20:24:43.935 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-10T20:24:44.126 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-10T20:24:44.154 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-10T20:24:44.192 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-10T20:24:44.556 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package radosgw. 2024-09-10T20:24:44.584 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../117-radosgw_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:44.622 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking radosgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:44.963 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package radosgw. 2024-09-10T20:24:44.991 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../117-radosgw_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:45.029 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking radosgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:45.484 INFO:teuthology.orchestra.run.smithi153.stdout:Selecting previously unselected package rbd-fuse. 2024-09-10T20:24:45.500 INFO:teuthology.orchestra.run.smithi153.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:45.542 INFO:teuthology.orchestra.run.smithi153.stdout:Unpacking rbd-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:45.838 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-10T20:24:46.049 INFO:teuthology.orchestra.run.smithi176.stdout:Selecting previously unselected package rbd-fuse. 2024-09-10T20:24:46.065 INFO:teuthology.orchestra.run.smithi176.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4858-gbad1f558-1jammy_amd64.deb ... 2024-09-10T20:24:46.097 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-10T20:24:46.107 INFO:teuthology.orchestra.run.smithi176.stdout:Unpacking rbd-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:46.223 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-10T20:24:46.404 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-10T20:24:46.483 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-10T20:24:46.653 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-10T20:24:46.779 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-10T20:24:46.829 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-10T20:24:47.075 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-10T20:24:47.252 INFO:teuthology.orchestra.run.smithi153.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-10T20:24:47.325 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:47.380 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-10T20:24:47.832 INFO:teuthology.orchestra.run.smithi176.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-10T20:24:47.841 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user cephadm....done 2024-09-10T20:24:47.906 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:48.005 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-10T20:24:48.282 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-10T20:24:48.447 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user cephadm....done 2024-09-10T20:24:48.524 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-10T20:24:48.595 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-10T20:24:48.651 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-10T20:24:48.855 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-10T20:24:48.910 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-10T20:24:49.123 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-10T20:24:49.153 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-10T20:24:49.249 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-10T20:24:49.253 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-10T20:24:49.500 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-10T20:24:49.564 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-10T20:24:49.759 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-10T20:24:49.885 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-10T20:24:49.932 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-10T20:24:50.187 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-10T20:24:50.202 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-10T20:24:50.468 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-10T20:24:50.539 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-10T20:24:50.711 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-10T20:24:50.798 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-10T20:24:50.945 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-10T20:24:51.074 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-10T20:24:51.196 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-10T20:24:51.351 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-10T20:24:51.485 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-argparse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:51.585 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-10T20:24:51.836 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-10T20:24:52.099 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-10T20:24:52.103 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-argparse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:52.199 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-10T20:24:52.300 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-10T20:24:52.346 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-10T20:24:52.428 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-10T20:24:52.447 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-10T20:24:52.552 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-10T20:24:52.573 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-10T20:24:52.699 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-10T20:24:52.820 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-10T20:24:52.825 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-10T20:24:53.076 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-10T20:24:53.204 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-10T20:24:53.235 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-10T20:24:53.298 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-10T20:24:53.399 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-10T20:24:53.441 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-10T20:24:53.474 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-10T20:24:53.537 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-10T20:24:53.664 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-10T20:24:53.750 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-10T20:24:53.868 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-10T20:24:54.023 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-10T20:24:54.127 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-10T20:24:54.149 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-10T20:24:54.386 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-10T20:24:54.433 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-10T20:24:54.696 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up zip (3.0-12build2) ... 2024-09-10T20:24:54.701 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-10T20:24:54.822 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-10T20:24:55.002 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up zip (3.0-12build2) ... 2024-09-10T20:24:55.128 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-10T20:24:55.515 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-10T20:24:55.791 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-10T20:24:55.813 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-10T20:24:55.901 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-10T20:24:56.026 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-10T20:24:56.056 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-10T20:24:56.173 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-10T20:24:56.299 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-10T20:24:56.319 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-10T20:24:56.600 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-10T20:24:56.695 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-10T20:24:56.955 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-10T20:24:56.993 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-10T20:24:57.244 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-10T20:24:57.331 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-10T20:24:57.599 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-10T20:24:57.629 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-10T20:24:57.914 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-10T20:24:57.925 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-10T20:24:58.177 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-10T20:24:58.235 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-10T20:24:58.286 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:58.482 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-10T20:24:58.570 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-10T20:24:58.609 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:24:58.910 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-10T20:24:58.913 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-10T20:24:59.227 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-10T20:25:00.085 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-10T20:25:00.329 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:25:00.425 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-10T20:25:00.471 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-10T20:25:00.676 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:25:00.731 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-10T20:25:00.810 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-10T20:25:00.832 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-10T20:25:00.933 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-10T20:25:01.070 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-10T20:25:01.171 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-10T20:25:01.226 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-10T20:25:01.272 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-10T20:25:01.469 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-10T20:25:01.573 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-10T20:25:01.703 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:25:01.824 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-10T20:25:01.829 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-10T20:25:02.075 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:25:02.122 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-10T20:25:02.201 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-10T20:25:02.381 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-10T20:25:02.503 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-10T20:25:02.641 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-10T20:25:02.753 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-10T20:25:02.900 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-10T20:25:02.996 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-10T20:25:03.143 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-10T20:25:03.231 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-10T20:25:03.394 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-10T20:25:03.482 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-10T20:25:03.495 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-10T20:25:03.749 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-10T20:25:03.770 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-10T20:25:03.876 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-10T20:25:03.897 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-10T20:25:04.160 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-10T20:25:04.198 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-10T20:25:04.286 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-10T20:25:04.499 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-10T20:25:04.578 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-10T20:25:04.750 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-10T20:25:04.854 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-10T20:25:04.876 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-10T20:25:04.993 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-10T20:25:05.065 INFO:teuthology.orchestra.run.smithi153.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-10T20:25:05.089 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-10T20:25:05.223 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-10T20:25:05.295 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-10T20:25:05.357 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-10T20:25:05.421 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-10T20:25:05.446 INFO:teuthology.orchestra.run.smithi176.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-10T20:25:05.684 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-10T20:25:05.810 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-10T20:25:05.823 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-10T20:25:06.141 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-10T20:25:06.220 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-10T20:25:06.266 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-10T20:25:06.462 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-10T20:25:06.526 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:25:06.563 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-10T20:25:06.652 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-10T20:25:06.814 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-10T20:25:06.940 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-10T20:25:06.961 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-10T20:25:07.250 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-10T20:25:07.352 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-10T20:25:07.551 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-10T20:25:07.655 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-10T20:25:07.781 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-10T20:25:07.827 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-10T20:25:07.953 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-10T20:25:08.065 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-10T20:25:08.229 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-10T20:25:08.418 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-10T20:25:08.557 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-10T20:25:09.253 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-10T20:25:09.253 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-10T20:25:09.253 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-10T20:25:09.253 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-10T20:25:09.381 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-10T20:25:09.469 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-10T20:25:09.734 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-10T20:25:09.838 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-10T20:25:09.843 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librados2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:09.952 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:09.964 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librados2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:10.061 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-10T20:25:10.090 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:10.216 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-10T20:25:10.406 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:10.532 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libradosstriper1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:10.552 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:10.658 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-10T20:25:10.678 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libradosstriper1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:10.796 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-10T20:25:10.935 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librbd1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:11.044 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-modules-core (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:11.056 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librbd1 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:11.145 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:11.182 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-modules-core (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:11.308 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:11.360 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T20:25:11.360 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T20:25:11.518 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T20:25:11.518 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-10T20:25:11.812 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up libcephfs-dev (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:11.972 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rados (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:12.045 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up libcephfs-dev (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:12.098 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up librgw2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:12.210 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rados (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:12.241 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-10T20:25:12.345 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up librgw2 (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:12.471 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-10T20:25:13.386 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rbd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:13.495 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up rbd-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:13.621 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-rgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:13.708 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rbd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:13.749 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up python3-cephfs (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:13.834 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up rbd-fuse (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:13.875 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:13.960 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-rgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:14.094 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up python3-cephfs (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:14.195 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-common (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:14.385 INFO:teuthology.orchestra.run.smithi153.stdout:Adding group ceph....done 2024-09-10T20:25:14.705 INFO:teuthology.orchestra.run.smithi176.stdout:Adding group ceph....done 2024-09-10T20:25:14.711 INFO:teuthology.orchestra.run.smithi153.stdout:Adding system user ceph....done 2024-09-10T20:25:14.864 INFO:teuthology.orchestra.run.smithi153.stdout:Setting system user ceph properties....done 2024-09-10T20:25:14.880 INFO:teuthology.orchestra.run.smithi153.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-10T20:25:15.015 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-10T20:25:15.048 INFO:teuthology.orchestra.run.smithi176.stdout:Adding system user ceph....done 2024-09-10T20:25:15.225 INFO:teuthology.orchestra.run.smithi176.stdout:Setting system user ceph properties....done 2024-09-10T20:25:15.237 INFO:teuthology.orchestra.run.smithi176.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-10T20:25:15.357 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-10T20:25:15.373 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-10T20:25:15.740 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-10T20:25:15.776 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-test (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:15.877 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up radosgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:16.188 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-test (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:16.297 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up radosgw (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:16.323 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:25:16.323 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:25:16.771 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-base (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:16.781 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:25:16.781 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-10T20:25:17.118 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-10T20:25:17.282 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-base (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:17.590 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mds (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:17.627 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-10T20:25:17.797 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T20:25:17.797 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T20:25:18.151 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mds (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:18.275 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:18.371 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T20:25:18.371 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-10T20:25:18.490 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T20:25:18.490 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T20:25:18.853 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:18.968 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-osd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:19.063 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T20:25:19.064 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-10T20:25:19.216 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T20:25:19.216 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T20:25:19.570 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-osd (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:19.703 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:19.827 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T20:25:19.827 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-10T20:25:19.828 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:19.980 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mon (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:20.189 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T20:25:20.189 INFO:teuthology.orchestra.run.smithi153.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T20:25:20.338 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:20.465 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:20.641 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mon (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:20.657 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:20.799 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:20.855 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T20:25:20.855 INFO:teuthology.orchestra.run.smithi176.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-10T20:25:20.916 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-mgr-dashboard (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:21.051 INFO:teuthology.orchestra.run.smithi153.stdout:Setting up ceph-volume (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:21.366 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-cephadm (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:21.404 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-10T20:25:21.501 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:21.504 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-10T20:25:21.610 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-mgr-dashboard (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:21.657 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-10T20:25:21.753 INFO:teuthology.orchestra.run.smithi176.stdout:Setting up ceph-volume (19.3.0-4858-gbad1f558-1jammy) ... 2024-09-10T20:25:22.275 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-10T20:25:22.384 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-10T20:25:22.536 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-10T20:25:22.784 INFO:teuthology.orchestra.run.smithi153.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-10T20:25:23.444 INFO:teuthology.orchestra.run.smithi153.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-10T20:25:23.450 DEBUG:teuthology.parallel:result is None 2024-09-10T20:25:23.661 INFO:teuthology.orchestra.run.smithi176.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-10T20:25:24.297 INFO:teuthology.orchestra.run.smithi176.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-10T20:25:24.302 DEBUG:teuthology.parallel:result is None 2024-09-10T20:25:24.302 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:25:24.448 DEBUG:teuthology.orchestra.run.smithi153:> dpkg-query -W -f '${Version}' ceph 2024-09-10T20:25:24.467 INFO:teuthology.orchestra.run.smithi153.stdout:19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:25:24.468 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:25:24.468 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858-gbad1f558-1jammy is installed. 2024-09-10T20:25:24.470 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:25:24.612 DEBUG:teuthology.orchestra.run.smithi176:> dpkg-query -W -f '${Version}' ceph 2024-09-10T20:25:24.631 INFO:teuthology.orchestra.run.smithi176.stdout:19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:25:24.631 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4858-gbad1f558-1jammy 2024-09-10T20:25:24.631 INFO:teuthology.task.install:The correct ceph version 19.3.0-4858-gbad1f558-1jammy is installed. 2024-09-10T20:25:24.633 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-10T20:25:24.633 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:24.634 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T20:25:24.648 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:24.648 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-10T20:25:24.684 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-10T20:25:24.685 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:24.685 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T20:25:24.702 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T20:25:24.754 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:24.754 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/daemon-helper 2024-09-10T20:25:24.767 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-10T20:25:24.820 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-10T20:25:24.821 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:24.821 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T20:25:24.833 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T20:25:24.886 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:24.886 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-10T20:25:24.898 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-10T20:25:24.951 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-10T20:25:24.952 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:24.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T20:25:24.968 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T20:25:25.023 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:25.023 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/usr/bin/stdin-killer 2024-09-10T20:25:25.035 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-10T20:25:25.087 INFO:teuthology.run_tasks:Running task ceph... 2024-09-10T20:25:25.169 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-10T20:25:25.170 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 777 /var/log/ceph 2024-09-10T20:25:25.172 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 777 /var/log/ceph 2024-09-10T20:25:25.185 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-10T20:25:25.185 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-10T20:25:25.228 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-10T20:25:25.240 INFO:tasks.ceph:Creating extra log directories... 2024-09-10T20:25:25.241 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-10T20:25:25.280 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-10T20:25:25.306 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-10T20:25:25.306 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-10T20:25:25.307 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561', 'branch': 'wip-bharath7-testing-2024-09-10-1409', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests}', 'email': 'skanta@redhat.com', 'first_in_suite': False, 'job_id': '7899561', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'wip-bharath7-testing-2024-09-10-1409'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(TOO_FEW_PGS\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-bharath7-testing-2024-09-10-1409', 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'}}, 'owner': 'scheduled_skanta@teuthology', 'priority': 99, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 5245, 'sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-bharath7-testing-2024-09-10-1409', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_bad1f558c327094897e4a32e99dd5cb4ae6ccd46/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46', 'targets': {'smithi153.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIDRdRZxOj9w6XChkKnhcfza9lnOD4JcUM6bRzevopJ3cfPbdQ1DGMIVYw1yfIlmUXMauFz33T5tQDtogNZZBso=', 'smithi176.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLKp1W2il4oXQDZt3QK2Nr5PTBtX+3C2ODJBk5ZDJPKgmIY6LIyZdiyriLTj7X8jBRz//2/wytqMBVD/qaj5O0U='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'store_thrash': True, 'thrash_delay': 1, 'thrash_many': True}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_15:30:27', 'tube': 'smithi', 'user': 'skanta', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-10T20:25:25.307 DEBUG:teuthology.orchestra.run.smithi153:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-10T20:25:25.336 DEBUG:teuthology.orchestra.run.smithi176:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-10T20:25:25.345 DEBUG:teuthology.orchestra.run.smithi153:> sudo install -d -m0777 -- /var/run/ceph 2024-09-10T20:25:25.384 DEBUG:teuthology.orchestra.run.smithi176:> sudo install -d -m0777 -- /var/run/ceph 2024-09-10T20:25:25.399 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:25.399 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T20:25:25.443 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T20:25:25.443 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_1 2024-09-10T20:25:25.491 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T20:25:25.491 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:25:25.491 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-09-10T20:25:25.491 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:25:25.492 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-10 20:19:59.239742302 +0000 2024-09-10T20:25:25.492 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-10 20:19:59.235742338 +0000 2024-09-10T20:25:25.492 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-10 20:19:59.235742338 +0000 2024-09-10T20:25:25.492 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-09-10T20:25:25.492 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T20:25:25.545 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-10T20:25:25.546 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-10T20:25:25.546 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000436405 s, 1.2 MB/s 2024-09-10T20:25:25.547 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T20:25:25.597 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_2 2024-09-10T20:25:25.647 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T20:25:25.647 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:25:25.647 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-09-10T20:25:25.647 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:25:25.648 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-10 20:19:59.951735942 +0000 2024-09-10T20:25:25.648 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-10 20:19:59.695738229 +0000 2024-09-10T20:25:25.648 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-10 20:19:59.695738229 +0000 2024-09-10T20:25:25.648 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-09-10T20:25:25.648 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T20:25:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-10T20:25:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-10T20:25:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000419587 s, 1.2 MB/s 2024-09-10T20:25:25.704 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T20:25:25.752 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_3 2024-09-10T20:25:25.799 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T20:25:25.799 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:25:25.800 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 920 Links: 1 2024-09-10T20:25:25.800 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:25:25.800 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-10 20:20:00.519730869 +0000 2024-09-10T20:25:25.800 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-10 20:20:00.243733334 +0000 2024-09-10T20:25:25.800 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-10 20:20:00.243733334 +0000 2024-09-10T20:25:25.800 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-09-10T20:25:25.800 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T20:25:25.853 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-10T20:25:25.853 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-10T20:25:25.853 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000308518 s, 1.7 MB/s 2024-09-10T20:25:25.854 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T20:25:25.904 DEBUG:teuthology.orchestra.run.smithi153:> stat /dev/vg_nvme/lv_4 2024-09-10T20:25:25.951 INFO:teuthology.orchestra.run.smithi153.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T20:25:25.951 INFO:teuthology.orchestra.run.smithi153.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:25:25.951 INFO:teuthology.orchestra.run.smithi153.stdout:Device: 5h/5d Inode: 935 Links: 1 2024-09-10T20:25:25.951 INFO:teuthology.orchestra.run.smithi153.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:25:25.951 INFO:teuthology.orchestra.run.smithi153.stdout:Access: 2024-09-10 20:20:00.991726652 +0000 2024-09-10T20:25:25.951 INFO:teuthology.orchestra.run.smithi153.stdout:Modify: 2024-09-10 20:20:00.751728797 +0000 2024-09-10T20:25:25.952 INFO:teuthology.orchestra.run.smithi153.stdout:Change: 2024-09-10 20:20:00.751728797 +0000 2024-09-10T20:25:25.952 INFO:teuthology.orchestra.run.smithi153.stdout: Birth: - 2024-09-10T20:25:25.952 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T20:25:26.006 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records in 2024-09-10T20:25:26.006 INFO:teuthology.orchestra.run.smithi153.stderr:1+0 records out 2024-09-10T20:25:26.006 INFO:teuthology.orchestra.run.smithi153.stderr:512 bytes copied, 0.000393674 s, 1.3 MB/s 2024-09-10T20:25:26.007 DEBUG:teuthology.orchestra.run.smithi153:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T20:25:26.057 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-10T20:25:26.057 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:26.057 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/scratch_devs of=/dev/stdout 2024-09-10T20:25:26.062 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-10T20:25:26.063 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_1 2024-09-10T20:25:26.109 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-10T20:25:26.109 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:25:26.109 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 874 Links: 1 2024-09-10T20:25:26.109 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:25:26.109 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-09-10 20:19:56.625167542 +0000 2024-09-10T20:25:26.109 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-09-10 20:19:56.381169721 +0000 2024-09-10T20:25:26.109 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-09-10 20:19:56.381169721 +0000 2024-09-10T20:25:26.109 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-09-10T20:25:26.110 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-10T20:25:26.164 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-09-10T20:25:26.164 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-09-10T20:25:26.164 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000455503 s, 1.1 MB/s 2024-09-10T20:25:26.165 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-10T20:25:26.214 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_2 2024-09-10T20:25:26.261 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-10T20:25:26.261 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:25:26.261 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 887 Links: 1 2024-09-10T20:25:26.261 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:25:26.261 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-09-10 20:19:57.137162969 +0000 2024-09-10T20:25:26.261 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-09-10 20:19:56.873165327 +0000 2024-09-10T20:25:26.261 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-09-10 20:19:56.873165327 +0000 2024-09-10T20:25:26.261 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-09-10T20:25:26.261 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-10T20:25:26.314 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-09-10T20:25:26.314 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-09-10T20:25:26.314 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000341287 s, 1.5 MB/s 2024-09-10T20:25:26.315 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-10T20:25:26.362 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_3 2024-09-10T20:25:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-10T20:25:26.409 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:25:26.409 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 908 Links: 1 2024-09-10T20:25:26.409 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:25:26.410 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-09-10 20:19:57.637158504 +0000 2024-09-10T20:25:26.410 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-09-10 20:19:57.377160826 +0000 2024-09-10T20:25:26.410 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-09-10 20:19:57.377160826 +0000 2024-09-10T20:25:26.410 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-09-10T20:25:26.410 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-10T20:25:26.463 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-09-10T20:25:26.463 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-09-10T20:25:26.463 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000334312 s, 1.5 MB/s 2024-09-10T20:25:26.464 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-10T20:25:26.514 DEBUG:teuthology.orchestra.run.smithi176:> stat /dev/vg_nvme/lv_4 2024-09-10T20:25:26.561 INFO:teuthology.orchestra.run.smithi176.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-10T20:25:26.561 INFO:teuthology.orchestra.run.smithi176.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-10T20:25:26.561 INFO:teuthology.orchestra.run.smithi176.stdout:Device: 5h/5d Inode: 921 Links: 1 2024-09-10T20:25:26.561 INFO:teuthology.orchestra.run.smithi176.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-10T20:25:26.561 INFO:teuthology.orchestra.run.smithi176.stdout:Access: 2024-09-10 20:19:58.141154001 +0000 2024-09-10T20:25:26.561 INFO:teuthology.orchestra.run.smithi176.stdout:Modify: 2024-09-10 20:19:57.893156216 +0000 2024-09-10T20:25:26.561 INFO:teuthology.orchestra.run.smithi176.stdout:Change: 2024-09-10 20:19:57.893156216 +0000 2024-09-10T20:25:26.562 INFO:teuthology.orchestra.run.smithi176.stdout: Birth: - 2024-09-10T20:25:26.562 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-10T20:25:26.619 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records in 2024-09-10T20:25:26.619 INFO:teuthology.orchestra.run.smithi176.stderr:1+0 records out 2024-09-10T20:25:26.619 INFO:teuthology.orchestra.run.smithi176.stderr:512 bytes copied, 0.000391177 s, 1.3 MB/s 2024-09-10T20:25:26.621 DEBUG:teuthology.orchestra.run.smithi176:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-10T20:25:26.670 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-10T20:25:26.670 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-10T20:25:26.670 INFO:tasks.ceph:Generating config... 2024-09-10T20:25:26.672 INFO:tasks.ceph:[client] debug monc = 20 2024-09-10T20:25:26.672 INFO:tasks.ceph:[client] debug ms = 1 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] debug ms = 1 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] debug rados = 20 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] ms bind msgr1 = False 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] ms bind msgr2 = True 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] ms inject delay max = 1 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2024-09-10T20:25:26.672 INFO:tasks.ceph:[global] ms inject delay type = mon 2024-09-10T20:25:26.673 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2024-09-10T20:25:26.673 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2024-09-10T20:25:26.673 INFO:tasks.ceph:[global] ms type = async 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mgr] debug monc = 10 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-10T20:25:26.673 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-10T20:25:26.674 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-10T20:25:26.674 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-10T20:25:26.674 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-10T20:25:26.674 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] bluestore write v2 random = True 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-10T20:25:26.674 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-10T20:25:26.675 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-10T20:25:26.676 INFO:tasks.ceph:Setting up mon.a... 2024-09-10T20:25:26.676 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-10T20:25:26.710 INFO:teuthology.orchestra.run.smithi153.stdout:creating /etc/ceph/ceph.keyring 2024-09-10T20:25:26.722 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-10T20:25:26.801 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T20:25:26.854 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.153'), ('mon.c', '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]'), ('mon.b', '172.21.15.176')] 2024-09-10T20:25:26.855 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],172.21.15.176', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2 random': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 5}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-10T20:25:26.856 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:26.856 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T20:25:26.903 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.153 --addv c '[v2:172.21.15.153:3301,v1:172.21.15.153:6790]' --add b 172.21.15.176 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:26.989 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: generated fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:setting min_mon_release = quincy 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:epoch 0 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:min_mon_release 17 (quincy) 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:election_strategy: 1 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:25:26.990 INFO:teuthology.orchestra.run.smithi153.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-10T20:25:26.992 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-10T20:25:27.040 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 8126fdd8-2c77-4da0-ba28-d5fdc347eec8... 2024-09-10T20:25:27.041 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-10T20:25:27.084 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-10T20:25:27.112 INFO:teuthology.orchestra.run.smithi153.stdout:[global] 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: chdir = "" 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: auth supported = cephx 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: filestore xattr use omap = true 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: mon clock drift allowed = 1.000 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: osd crush chooseleaf type = 0 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: auth debug = true 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.113 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on old message = true 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: ms die on bug = true 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: mon pg warn max object skew = 0 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: # disable pg_autoscaler by default for new pools 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default size = 2 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow primary affinity = true 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd allow pg remap = true 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on legacy crush tunables = false 2024-09-10T20:25:27.114 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on crush straw calc version zero = false 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on no sortbitwise = false 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on osd down out interval zero = false 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on too few osds = false 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: mon_allow_pool_size_one = true 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: osd default data pool replay window = 5 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: mon allow pool delete = true 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.115 INFO:teuthology.orchestra.run.smithi153.stdout: mon cluster log file level = debug 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: debug asserts on shutdown = true 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: mon health detail to clog = false 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: mon host = "172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],172.21.15.176" 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: debug objecter = 20 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: debug rados = 20 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: mon client directed command retry = 5 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: mon election default strategy = 1 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: ms bind msgr1 = False 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: ms bind msgr2 = True 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay max = 1 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay probability = 0.005 2024-09-10T20:25:27.116 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject delay type = mon 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject internal delays = 0.002 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: ms inject socket failures = 2500 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: ms type = async 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: fsid = 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout:[osd] 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: osd journal size = 100 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub load threshold = 5.0 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: osd scrub max interval = 600 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock profile = high_recovery_ops 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: osd recover clone overlap = true 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: osd recovery max chunk = 1048576 2024-09-10T20:25:27.117 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug shutdown = true 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug op order = true 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify stray on activate = true 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug trim objects = true 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: osd open classes on start = true 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug pg log writeout = true 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: osd deep scrub update digest min age = 30 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: osd map max advance = 10 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: journal zero on create = true 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.118 INFO:teuthology.orchestra.run.smithi153.stdout: filestore ondisk finisher threads = 3 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: filestore apply finisher threads = 3 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: bdev debug aio = true 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug misdirected ops = true 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore block size = 96636764160 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression algorithm = snappy 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore compression mode = aggressive 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore fsck on mount = True 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore write v2 random = True 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: bluestore zero block detection = True 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluefs = 20 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: debug bluestore = 20 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-10T20:25:27.119 INFO:teuthology.orchestra.run.smithi153.stdout: debug osd = 20 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: debug rocksdb = 10 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd backfillfull_ratio = 0.85 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd full ratio = 0.9 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd nearfull ratio = 0.8 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: osd class default list = * 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: osd class load list = * 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify cached snaps = True 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: osd debug verify missing on start = True 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: osd failsafe full ratio = 0.95 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: osd mclock override recovery settings = True 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: osd objectstore = bluestore 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue = debug_random 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: osd op queue cut off = debug_random 2024-09-10T20:25:27.120 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr] 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: debug mgr = 20 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: mgr/telemetry/nag = false 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 10 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout:[mon] 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: debug mon = 20 2024-09-10T20:25:27.121 INFO:teuthology.orchestra.run.smithi153.stdout: debug paxos = 20 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: debug auth = 20 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: mon data avail warn = 5 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: mon mgr mkfs grace = 240 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd reporter subtree level = osd 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: mon osd prime pg temp = true 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: auth mon ticket ttl = 660 # 11m 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: auth service ticket ttl = 240 # 4m 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: # don't complain about insecure global_id in the test suite 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-10T20:25:27.122 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: # 1m isn't quite enough 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon_down_mkfs_grace = 2m 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon_warn_on_filestore_osds = false 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon min osdmap epochs = 25 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune interval = 2 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune min = 15 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon osdmap full prune txsize = 2 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject crc mismatch = 0.01 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub inject missing keys = 0.05 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon scrub interval = 300 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: mon warn on pool no app = False 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: paxos service trim min = 5 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:25:27.123 INFO:teuthology.orchestra.run.smithi153.stdout:[client] 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout: rgw cache enabled = true 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable ops log = true 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout: rgw enable usage log = true 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout: debug monc = 20 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout: debug ms = 1 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.a] 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.c] 2024-09-10T20:25:27.124 INFO:teuthology.orchestra.run.smithi153.stdout:[mon.b] 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout:[global] 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: chdir = "" 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: auth supported = cephx 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: filestore xattr use omap = true 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: mon clock drift allowed = 1.000 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: osd crush chooseleaf type = 0 2024-09-10T20:25:27.126 INFO:teuthology.orchestra.run.smithi176.stdout: auth debug = true 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on old message = true 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: ms die on bug = true 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: mon pg warn max object skew = 0 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: # disable pg_autoscaler by default for new pools 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default size = 2 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow primary affinity = true 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd allow pg remap = true 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on legacy crush tunables = false 2024-09-10T20:25:27.127 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on crush straw calc version zero = false 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on no sortbitwise = false 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on osd down out interval zero = false 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on too few osds = false 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: mon_allow_pool_size_one = true 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: osd default data pool replay window = 5 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: mon allow pool delete = true 2024-09-10T20:25:27.128 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: mon cluster log file level = debug 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: debug asserts on shutdown = true 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: mon health detail to clog = false 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: mon host = "172.21.15.153,[v2:172.21.15.153:3301,v1:172.21.15.153:6790],172.21.15.176" 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: debug objecter = 20 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: debug rados = 20 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: mon client directed command retry = 5 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: mon election default strategy = 1 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: ms bind msgr1 = False 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: ms bind msgr2 = True 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject delay max = 1 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject delay probability = 0.005 2024-09-10T20:25:27.129 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject delay type = mon 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject internal delays = 0.002 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: ms inject socket failures = 2500 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: ms type = async 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: fsid = 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout:[osd] 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: osd journal size = 100 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub load threshold = 5.0 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: osd scrub max interval = 600 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock profile = high_recovery_ops 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: osd recover clone overlap = true 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: osd recovery max chunk = 1048576 2024-09-10T20:25:27.130 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug shutdown = true 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug op order = true 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify stray on activate = true 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug trim objects = true 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: osd open classes on start = true 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug pg log writeout = true 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: osd deep scrub update digest min age = 30 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: osd map max advance = 10 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: journal zero on create = true 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.131 INFO:teuthology.orchestra.run.smithi176.stdout: filestore ondisk finisher threads = 3 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: filestore apply finisher threads = 3 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: bdev debug aio = true 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug misdirected ops = true 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore block size = 96636764160 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore compression algorithm = snappy 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore compression mode = aggressive 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore fsck on mount = True 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore write v2 random = True 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: bluestore zero block detection = True 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluefs = 20 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: debug bluestore = 20 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: debug osd = 20 2024-09-10T20:25:27.132 INFO:teuthology.orchestra.run.smithi176.stdout: debug rocksdb = 10 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd backfillfull_ratio = 0.85 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd full ratio = 0.9 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd nearfull ratio = 0.8 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: osd class default list = * 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: osd class load list = * 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify cached snaps = True 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: osd debug verify missing on start = True 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: osd failsafe full ratio = 0.95 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: osd mclock override recovery settings = True 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: osd objectstore = bluestore 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: osd op queue = debug_random 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: osd op queue cut off = debug_random 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.133 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr] 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: debug mgr = 20 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: mgr/telemetry/nag = false 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: debug monc = 10 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout:[mon] 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: debug mon = 20 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: debug paxos = 20 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: debug auth = 20 2024-09-10T20:25:27.134 INFO:teuthology.orchestra.run.smithi176.stdout: mon data avail warn = 5 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: mon mgr mkfs grace = 240 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min pgs per osd = 4 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd reporter subtree level = osd 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: mon osd prime pg temp = true 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: mon reweight min bytes per osd = 10 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: auth mon ticket ttl = 660 # 11m 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: auth service ticket ttl = 240 # 4m 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: # don't complain about insecure global_id in the test suite 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-10T20:25:27.135 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: # 1m isn't quite enough 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon_down_mkfs_grace = 2m 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon_warn_on_filestore_osds = false 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon min osdmap epochs = 25 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune interval = 2 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune min = 15 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon osdmap full prune txsize = 2 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon scrub inject crc mismatch = 0.01 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon scrub inject missing keys = 0.05 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon scrub interval = 300 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: mon warn on pool no app = False 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: paxos service trim min = 5 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout: 2024-09-10T20:25:27.136 INFO:teuthology.orchestra.run.smithi176.stdout:[client] 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout: rgw cache enabled = true 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable ops log = true 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout: rgw enable usage log = true 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout: debug monc = 20 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout: debug ms = 1 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.a] 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.c] 2024-09-10T20:25:27.137 INFO:teuthology.orchestra.run.smithi176.stdout:[mon.b] 2024-09-10T20:25:27.137 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-10T20:25:27.138 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-10T20:25:27.221 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-10T20:25:27.221 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:27.221 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-10T20:25:27.226 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:27.226 DEBUG:teuthology.orchestra.run.smithi153:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-10T20:25:27.275 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi153.front.sepia.ceph.com 2024-09-10T20:25:27.275 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:27.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-10T20:25:27.275 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T20:25:27.339 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:27.339 DEBUG:teuthology.orchestra.run.smithi153:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:27.387 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi176.front.sepia.ceph.com 2024-09-10T20:25:27.387 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:27.387 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-10T20:25:27.387 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-10T20:25:27.409 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:27.409 DEBUG:teuthology.orchestra.run.smithi176:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:27.457 INFO:tasks.ceph:Setting up mon nodes... 2024-09-10T20:25:27.457 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-10T20:25:27.458 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T20:25:27.543 INFO:teuthology.orchestra.run.smithi176.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-10T20:25:27.566 INFO:tasks.ceph:Setting up mds nodes... 2024-09-10T20:25:27.566 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-10T20:25:27.566 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-10T20:25:27.643 INFO:teuthology.orchestra.run.smithi176.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-10T20:25:27.654 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-10T20:25:27.654 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi153.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi176.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-10T20:25:27.654 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-10T20:25:27.668 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-10T20:25:27.668 INFO:tasks.ceph:role: osd.0 2024-09-10T20:25:27.668 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-10T20:25:27.668 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:25:27.731 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:25:28.196 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-10T20:25:28.198 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-10T20:25:28.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-10T20:25:28.293 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-10T20:25:28.301 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2024-09-10T20:25:28.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:25:28.302 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-10T20:25:28.359 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-10T20:25:28.360 INFO:tasks.ceph:role: osd.1 2024-09-10T20:25:28.360 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-10T20:25:28.360 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-10T20:25:28.423 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:25:28.423 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:25:28.423 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:25:28.423 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-10T20:25:28.424 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:25:28.424 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:25:28.424 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:25:28.424 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-10T20:25:28.424 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:25:28.424 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:25:28.898 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-10T20:25:28.900 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-10T20:25:28.900 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-10T20:25:29.061 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-10T20:25:29.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:25:29.073 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2024-09-10T20:25:29.073 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-10T20:25:29.131 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-10T20:25:29.131 INFO:tasks.ceph:role: osd.2 2024-09-10T20:25:29.131 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-10T20:25:29.131 DEBUG:teuthology.orchestra.run.smithi153:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:25:29.194 INFO:teuthology.orchestra.run.smithi153.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:25:29.677 INFO:teuthology.orchestra.run.smithi153.stdout:Discarding blocks...Done. 2024-09-10T20:25:29.677 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi153.front.sepia.ceph.com -o noatime 2024-09-10T20:25:29.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-10T20:25:29.816 DEBUG:teuthology.orchestra.run.smithi153:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-10T20:25:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:25:29.828 INFO:teuthology.orchestra.run.smithi153.stderr:sudo: /sbin/restorecon: command not found 2024-09-10T20:25:29.829 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:29.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:29.938+0000 7f29394db7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-10T20:25:29.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:29.938+0000 7f29394db7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-10T20:25:29.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:29.938+0000 7f29394db7c0 -1 bdev(0x55edfab8d800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-10T20:25:29.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:29.938+0000 7f29394db7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-10T20:25:33.083 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-10T20:25:33.097 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:33.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:33.170+0000 7fd360c4d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-10T20:25:33.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:33.170+0000 7fd360c4d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-10T20:25:33.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:33.170+0000 7fd360c4d7c0 -1 bdev(0x559cc3d65800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-10T20:25:33.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:33.170+0000 7fd360c4d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-10T20:25:36.363 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-10T20:25:36.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:36.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:36.494+0000 7f315fa2d7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-10T20:25:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:36.494+0000 7f315fa2d7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-10T20:25:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:36.494+0000 7f315fa2d7c0 -1 bdev(0x5585f2b13800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-10T20:25:36.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:36.494+0000 7f315fa2d7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-10T20:25:39.632 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-10T20:25:39.688 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-10T20:25:39.703 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-10T20:25:39.703 INFO:tasks.ceph:role: osd.3 2024-09-10T20:25:39.703 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-10T20:25:39.703 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:25:39.764 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:25:40.224 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-09-10T20:25:40.226 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-09-10T20:25:40.226 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-10T20:25:40.320 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-10T20:25:40.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:25:40.330 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-09-10T20:25:40.330 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-10T20:25:40.382 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-10T20:25:40.382 INFO:tasks.ceph:role: osd.4 2024-09-10T20:25:40.382 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-10T20:25:40.382 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-10T20:25:40.446 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:25:40.446 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:25:40.446 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:25:40.446 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-10T20:25:40.446 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:25:40.446 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:25:40.446 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:25:40.446 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-10T20:25:40.446 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:25:40.447 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:25:40.916 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-09-10T20:25:40.918 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-09-10T20:25:40.919 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-10T20:25:41.069 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-10T20:25:41.080 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-09-10T20:25:41.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:25:41.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-10T20:25:41.136 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-10T20:25:41.136 INFO:tasks.ceph:role: osd.5 2024-09-10T20:25:41.136 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-10T20:25:41.136 DEBUG:teuthology.orchestra.run.smithi176:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-10T20:25:41.200 INFO:teuthology.orchestra.run.smithi176.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-10T20:25:41.200 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-10T20:25:41.200 INFO:teuthology.orchestra.run.smithi176.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-10T20:25:41.200 INFO:teuthology.orchestra.run.smithi176.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-10T20:25:41.201 INFO:teuthology.orchestra.run.smithi176.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-10T20:25:41.201 INFO:teuthology.orchestra.run.smithi176.stdout: = sunit=0 swidth=0 blks 2024-09-10T20:25:41.201 INFO:teuthology.orchestra.run.smithi176.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-10T20:25:41.201 INFO:teuthology.orchestra.run.smithi176.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-10T20:25:41.201 INFO:teuthology.orchestra.run.smithi176.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-10T20:25:41.201 INFO:teuthology.orchestra.run.smithi176.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-10T20:25:41.670 INFO:teuthology.orchestra.run.smithi176.stdout:Discarding blocks...Done. 2024-09-10T20:25:41.672 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi176.front.sepia.ceph.com -o noatime 2024-09-10T20:25:41.672 DEBUG:teuthology.orchestra.run.smithi176:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-10T20:25:41.834 DEBUG:teuthology.orchestra.run.smithi176:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-10T20:25:41.846 INFO:teuthology.orchestra.run.smithi176.stderr:sudo: /sbin/restorecon: command not found 2024-09-10T20:25:41.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:25:41.847 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:41.958 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:41.953+0000 7f6b5272f7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-10T20:25:41.960 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:41.957+0000 7f6b5272f7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-10T20:25:41.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:41.957+0000 7f6b5272f7c0 -1 bdev(0x564a8ca69800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-10T20:25:41.961 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:41.957+0000 7f6b5272f7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-10T20:25:45.113 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-10T20:25:45.127 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:45.207 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:45.201+0000 7fba8188a7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-10T20:25:45.208 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:45.201+0000 7fba8188a7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-10T20:25:45.208 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:45.201+0000 7fba8188a7c0 -1 bdev(0x55f6f77ef800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-10T20:25:45.208 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:45.205+0000 7fba8188a7c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-10T20:25:48.336 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-10T20:25:48.393 DEBUG:teuthology.orchestra.run.smithi176:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:48.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:48.469+0000 7f4b354c67c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-10T20:25:48.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:48.469+0000 7f4b354c67c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-10T20:25:48.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:48.469+0000 7f4b354c67c0 -1 bdev(0x563485fc7800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-10T20:25:48.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:25:48.469+0000 7f4b354c67c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-10T20:25:51.619 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-10T20:25:51.673 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-10T20:25:51.674 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:51.674 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-10T20:25:51.688 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:51.688 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-10T20:25:51.741 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:25:51.742 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-10T20:25:51.794 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:51.794 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-10T20:25:51.807 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:51.808 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-10T20:25:51.861 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:51.861 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-10T20:25:51.917 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:51.917 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-10T20:25:51.972 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:25:51.972 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-10T20:25:52.021 INFO:tasks.ceph:Adding keys to all mons... 2024-09-10T20:25:52.021 DEBUG:teuthology.orchestra.run.smithi153:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-10T20:25:52.023 DEBUG:teuthology.orchestra.run.smithi176:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-10T20:25:52.067 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.0] 2024-09-10T20:25:52.067 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA5q+Bmk6IoOBAAf7SSs3lc0MIHmJN/pDZ7fA== 2024-09-10T20:25:52.067 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.1] 2024-09-10T20:25:52.067 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA9q+BmiMaLChAA5gwrUHVzeM/l2Rmdf+PVpg== 2024-09-10T20:25:52.067 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.2] 2024-09-10T20:25:52.067 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBAq+BmpcnEHRAARHYTgOOaZmZG7+CM0muN2A== 2024-09-10T20:25:52.067 INFO:teuthology.orchestra.run.smithi153.stdout:[mgr.x] 2024-09-10T20:25:52.067 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA3q+BmE7FcIBAAfOZQh4qXWBcqvLWDsmSxHg== 2024-09-10T20:25:52.067 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.3] 2024-09-10T20:25:52.068 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBFq+BmVRIXORAAkhcFsM6Lm/PKxv4ZokmAqA== 2024-09-10T20:25:52.068 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.4] 2024-09-10T20:25:52.068 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBJq+Bm3ZJVDBAA9y5A06dc2XrrVe6DowQ7ew== 2024-09-10T20:25:52.068 INFO:teuthology.orchestra.run.smithi153.stdout:[osd.5] 2024-09-10T20:25:52.068 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQBMq+BmaONMHBAAFmCs4VMhmpi64qm7qFDd7w== 2024-09-10T20:25:52.068 INFO:teuthology.orchestra.run.smithi153.stdout:[client.0] 2024-09-10T20:25:52.068 INFO:teuthology.orchestra.run.smithi153.stdout: key = AQA3q+BmDhtNJhAA7OoMISb1FO/A1r5xXYQgsw== 2024-09-10T20:25:52.075 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.0] 2024-09-10T20:25:52.075 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA5q+Bmk6IoOBAAf7SSs3lc0MIHmJN/pDZ7fA== 2024-09-10T20:25:52.075 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.1] 2024-09-10T20:25:52.075 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA9q+BmiMaLChAA5gwrUHVzeM/l2Rmdf+PVpg== 2024-09-10T20:25:52.075 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.2] 2024-09-10T20:25:52.075 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQBAq+BmpcnEHRAARHYTgOOaZmZG7+CM0muN2A== 2024-09-10T20:25:52.075 INFO:teuthology.orchestra.run.smithi176.stdout:[mgr.x] 2024-09-10T20:25:52.076 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA3q+BmE7FcIBAAfOZQh4qXWBcqvLWDsmSxHg== 2024-09-10T20:25:52.076 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.3] 2024-09-10T20:25:52.076 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQBFq+BmVRIXORAAkhcFsM6Lm/PKxv4ZokmAqA== 2024-09-10T20:25:52.076 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.4] 2024-09-10T20:25:52.076 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQBJq+Bm3ZJVDBAA9y5A06dc2XrrVe6DowQ7ew== 2024-09-10T20:25:52.076 INFO:teuthology.orchestra.run.smithi176.stdout:[osd.5] 2024-09-10T20:25:52.076 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQBMq+BmaONMHBAAFmCs4VMhmpi64qm7qFDd7w== 2024-09-10T20:25:52.076 INFO:teuthology.orchestra.run.smithi176.stdout:[client.0] 2024-09-10T20:25:52.076 INFO:teuthology.orchestra.run.smithi176.stdout: key = AQA3q+BmDhtNJhAA7OoMISb1FO/A1r5xXYQgsw== 2024-09-10T20:25:52.077 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.112 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.167 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.252 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.255 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.338 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T20:25:52.340 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-10T20:25:52.419 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.423 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.499 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.508 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.584 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.586 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-10T20:25:52.668 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:25:52.670 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-10T20:25:52.756 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-10T20:25:52.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-10T20:25:52.768 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T20:25:52.860 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-10T20:25:52.871 DEBUG:teuthology.orchestra.run.smithi153:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-10T20:25:52.924 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T20:25:53.014 DEBUG:teuthology.orchestra.run.smithi153:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-10T20:25:53.068 DEBUG:teuthology.orchestra.run.smithi176:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-10T20:25:53.082 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-10T20:25:53.178 DEBUG:teuthology.orchestra.run.smithi176:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-10T20:25:53.190 DEBUG:teuthology.orchestra.run.smithi153:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:53.192 DEBUG:teuthology.orchestra.run.smithi176:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-10T20:25:53.236 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-10T20:25:53.236 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T20:25:53.237 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T20:25:53.240 INFO:tasks.ceph.mon.a:Started 2024-09-10T20:25:53.240 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T20:25:53.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T20:25:53.242 INFO:tasks.ceph.mon.c:Started 2024-09-10T20:25:53.242 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-10T20:25:53.242 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-10T20:25:53.282 INFO:tasks.ceph.mon.b:Started 2024-09-10T20:25:53.282 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-10T20:25:53.282 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-10T20:25:53.282 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-10T20:25:53.284 INFO:tasks.ceph.mgr.x:Started 2024-09-10T20:25:53.285 DEBUG:tasks.ceph:set 0 configs 2024-09-10T20:25:53.285 INFO:tasks.ceph:Setting crush tunables to default 2024-09-10T20:25:53.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd crush tunables default 2024-09-10T20:25:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient: get_monmap_and_config 2024-09-10T20:25:53.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient: build_initial_monmap 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient: monmap: 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:53.402763+0000 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:53.402763+0000 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:25:53.404 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 1 Processor -- start 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 1 -- start start 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient: init 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient: _reopen_session rank -1 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient: _add_conns ranks=[4,0,3,2,1] 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient(hunting): picked mon.noname-c con 0x7f18c00a5a40 addr v2:172.21.15.176:3300/0 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f18c00a5e20 addr v1:172.21.15.153:6789/0 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient(hunting): picked mon.noname-a con 0x7f18c013db90 addr v2:172.21.15.153:3300/0 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c00a6b70 con 0x7f18c00a5e20 2024-09-10T20:25:53.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f18c00708a0 con 0x7f18c013db90 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f18c00a26b0 con 0x7f18c00a5a40 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient(hunting): _renew_subs 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18c4b38640 10 monclient(hunting): authenticate will time out at 913.542226s 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59808/0 (socket says 172.21.15.153:59808) 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 learned_addr learned my addr 172.21.15.153:0/4074260930 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bdd9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f18c00a5e20 0x7f18c00a62b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45920/0 (socket says 172.21.15.153:45920) 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 10 monclient(hunting): get_auth_request con 0x7f18c013db90 auth_method 0 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:25:53.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18c013ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18c013ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18c00a5a40 auth_method 0 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 msgr2=0x7f18c00a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 msgr2=0x7f18c00a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.398+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:25:53.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.402+0000 7f18bdd9b640 1 -- 172.21.15.153:0/4074260930 >> v1:172.21.15.153:6789/0 conn(0x7f18c00a5e20 legacy=0x7f18c00a62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:25:53.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:53.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18c00a5a40 auth_method 0 2024-09-10T20:25:53.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:53.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:53.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18be59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:53.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 msgr2=0x7f18c00a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:25:53.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 msgr2=0x7f18c00a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:53.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:53.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:25:53.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:53.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18bed9d640 10 monclient(hunting): get_auth_request con 0x7f18c013db90 auth_method 0 2024-09-10T20:25:53.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:53.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:53.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18bed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:53.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18c013ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:25:53.609 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18c013ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:53.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:53.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:53.602+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:25:54.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:54.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18bed9d640 10 monclient(hunting): get_auth_request con 0x7f18c013db90 auth_method 0 2024-09-10T20:25:54.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:54.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:54.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18bed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:54.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18c013ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:25:54.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18c013ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:54.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:54.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:25:54.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:54.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18c00a5a40 auth_method 0 2024-09-10T20:25:54.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:54.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:54.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18be59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:54.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 msgr2=0x7f18c00a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:25:54.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 msgr2=0x7f18c00a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:54.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:54.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.006+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:25:54.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:54.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18bed9d640 10 monclient(hunting): get_auth_request con 0x7f18c013db90 auth_method 0 2024-09-10T20:25:54.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:54.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:54.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18bed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18c013ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18c013ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18bed9d640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18c00a5a40 auth_method 0 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18be59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 msgr2=0x7f18c00a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 msgr2=0x7f18c00a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:54.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:54.810+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 10 monclient(hunting): tick 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 1 monclient(hunting): continuing hunt 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 msgr2=0x7f18c00a3930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.176:3300/0 conn(0x7f18c00a5a40 0x7f18c00a3930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18c013ff80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18c013ff80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 >> v1:172.21.15.153:6789/0 conn(0x7f18c00a5e20 legacy=0x7f18c00a62b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 10 monclient: _add_conns ranks=[2,1] 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:25:56.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f18a0003720 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18a0006280 addr v1:172.21.15.176:6789/0 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c00705f0 con 0x7f18a0006280 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f18c00708a0 con 0x7f18a0003720 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18b7fff640 10 monclient(hunting): _renew_subs 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18a0003720 auth_method 0 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:56.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:56.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:25:56.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.402+0000 7f18bdd9b640 1 -- 172.21.15.153:0/4074260930 >> v1:172.21.15.176:6789/0 conn(0x7f18a0006280 legacy=0x7f18a0008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:25:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.606+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.606+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18a0003720 auth_method 0 2024-09-10T20:25:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.606+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.606+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.606+0000 7f18be59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.606+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:25:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.606+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.606+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:56.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:56.606+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:25:57.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.010+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:57.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.010+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18a0003720 auth_method 0 2024-09-10T20:25:57.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.010+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:57.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.010+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:57.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.010+0000 7f18be59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:57.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.010+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:25:57.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.010+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:57.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.010+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:57.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.010+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:25:57.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.814+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:57.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.814+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18a0003720 auth_method 0 2024-09-10T20:25:57.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.814+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:57.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.814+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:57.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.814+0000 7f18be59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:57.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.814+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:25:57.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.814+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:57.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.814+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:57.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:57.814+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:25:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient(hunting): tick 2024-09-10T20:25:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient(hunting): _check_auth_tickets 2024-09-10T20:25:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 1 monclient(hunting): continuing hunt 2024-09-10T20:25:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T20:25:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:25:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:25:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 >> v1:172.21.15.176:6789/0 conn(0x7f18a0006280 legacy=0x7f18a0008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:25:59.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18a000a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient(hunting): picked mon.noname-a con 0x7f18c013db90 addr v2:172.21.15.153:3300/0 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 reap_dead start 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient(hunting): picked mon.noname-b con 0x7f18a0003720 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f18a000a720 addr v1:172.21.15.176:6789/0 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18a000b0f0 con 0x7f18a000a720 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f18c00708a0 con 0x7f18c013db90 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 1 -- 172.21.15.153:0/4074260930 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f18c00a26b0 con 0x7f18a0003720 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18b7fff640 10 monclient(hunting): _renew_subs 2024-09-10T20:25:59.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18a000a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18c013db90 auth_method 0 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bdd9b640 10 monclient(hunting): get_auth_request con 0x7f18a0003720 auth_method 0 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18a000a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18a000a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18a000a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:59.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bdd9b640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:25:59.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bdd9b640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:59.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:59.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18a000a1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:25:59.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:25:59.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.402+0000 7f18bed9d640 1 -- 172.21.15.153:0/4074260930 >> v1:172.21.15.176:6789/0 conn(0x7f18a000a720 legacy=0x7f18a000f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:25:59.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18a000a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18c013db90 auth_method 0 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18be59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18a000a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 msgr2=0x7f18a000a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18a000a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18bdd9b640 10 monclient(hunting): get_auth_request con 0x7f18a0003720 auth_method 0 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18a000a1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:25:59.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:25:59.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:59.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18bdd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:25:59.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18bdd9b640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:25:59.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18bdd9b640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:25:59.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:25:59.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.606+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:25:59.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.946+0000 7f18bd59a640 1 -- 172.21.15.153:0/4074260930 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698008225 0 0) 0x7f18a000b0f0 con 0x7f18a000a720 2024-09-10T20:25:59.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.946+0000 7f18bd59a640 10 monclient(hunting): _negotiate 2024-09-10T20:25:59.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.946+0000 7f18bd59a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:25:59.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.946+0000 7f18bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:25:59.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.946+0000 7f18bd59a640 10 monclient(hunting): my global_id is 4102 2024-09-10T20:25:59.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:25:59.946+0000 7f18bd59a640 1 -- 172.21.15.153:0/4074260930 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18ac003530 con 0x7f18a000a720 2024-09-10T20:26:00.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> v2:172.21.15.153:3300/0 conn(0x7f18c013db90 0x7f18a000a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18c013db90 auth_method 0 2024-09-10T20:26:00.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:00.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:00.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18bdd9b640 10 monclient(hunting): get_auth_request con 0x7f18a0003720 auth_method 0 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18bdd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient(hunting): handle_auth_done global_id 4098 payload 274 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 msgr2=0x7f18a0005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> v1:172.21.15.176:6789/0 conn(0x7f18a000a720 legacy=0x7f18a000f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient: _finish_hunting 0 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 1 monclient: found mon.noname-a 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:00.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:00.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.010+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a000add0 con 0x7f18c013db90 2024-09-10T20:26:00.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18be59c640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c013db90 msgr2=0x7f18a000a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:00.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18be59c640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c013db90 0x7f18a000a1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f18b8000ee0 tx=0x7f18b80071a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c536ee1b1dbbff96 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:00.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 1 -- 172.21.15.153:0/4074260930 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b800f040 con 0x7f18c013db90 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 10 monclient: dump: 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:00.025 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 10 monclient: _finish_auth 0 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 10 monclient: _check_auth_tickets 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 1 -- 172.21.15.153:0/4074260930 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b801b0f0 con 0x7f18c013db90 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 5 monclient: authenticate success, global_id 4098 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 4 set_mon_vals no callback set 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 1 -- 172.21.15.153:0/4074260930 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b8004020 con 0x7f18c013db90 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18bd59a640 10 monclient: dump: 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:00.026 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:00.027 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.027 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.027 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:00.027 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:00.027 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:00.027 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:00.027 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:00.027 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:00.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 10 monclient: shutdown 2024-09-10T20:26:00.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:00.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 1 -- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c013db90 msgr2=0x7f18a000a1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:00.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c013db90 0x7f18a000a1c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f18b8000ee0 tx=0x7f18b80071a0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 1 -- 172.21.15.153:0/4074260930 shutdown_connections 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18a0003720 0x7f18a0005b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 1 --2- 172.21.15.153:0/4074260930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18c013db90 0x7f18a000a1c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 1 -- 172.21.15.153:0/4074260930 >> 172.21.15.153:0/4074260930 conn(0x7f18c00ab980 msgr2=0x7f18c00a7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 1 -- 172.21.15.153:0/4074260930 shutdown_connections 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.018+0000 7f18c4b38640 1 -- 172.21.15.153:0/4074260930 wait complete. 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient: build_initial_monmap 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient: monmap: 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:00.032 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 librados: starting msgr at 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 librados: starting objecter 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 Processor -- start 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 -- start start 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 librados: setting wanted keys 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 librados: calling monclient init 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient: init 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:00.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a0003720 0x7f18c009fa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient(hunting): picked mon.noname-b con 0x7f18a0003720 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a000a720 0x7f18c009ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient(hunting): picked mon.noname-a con 0x7f18a000a720 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c013db90 0x7f18c00a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient(hunting): picked mon.noname-c con 0x7f18c013db90 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f18c0139270 con 0x7f18a000a720 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f18c01390f0 con 0x7f18a0003720 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f18c01393f0 con 0x7f18c013db90 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:00.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18c4b38640 10 monclient(hunting): authenticate will time out at 920.166509s 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a000a720 0x7f18c009ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c013db90 0x7f18c00a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a000a720 0x7f18c009ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:59848/0 (socket says 172.21.15.153:59848) 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c013db90 0x7f18c00a04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47024/0 (socket says 172.21.15.153:47024) 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 1 -- 172.21.15.153:0/2974707000 learned_addr learned my addr 172.21.15.153:0/2974707000 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient(hunting): get_auth_request con 0x7f18a000a720 auth_method 0 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 10 monclient(hunting): get_auth_request con 0x7f18c013db90 auth_method 0 2024-09-10T20:26:00.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18be59c640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a0003720 0x7f18c009fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18be59c640 10 monclient(hunting): get_auth_request con 0x7f18a0003720 auth_method 0 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18be59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18be59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18be59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:00.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient(hunting): handle_auth_done global_id 4101 payload 824 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 1 -- 172.21.15.153:0/2974707000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c013db90 msgr2=0x7f18c00a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c013db90 0x7f18c00a04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 1 -- 172.21.15.153:0/2974707000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a0003720 msgr2=0x7f18c009fa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a0003720 0x7f18c009fa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient: _finish_hunting 0 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 1 monclient: found mon.noname-a 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bdd9b640 1 -- 172.21.15.153:0/2974707000 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c012b8f0 con 0x7f18a000a720 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18bed9d640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c013db90 0x7f18c00a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:00.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.022+0000 7f18be59c640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a0003720 0x7f18c009fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:26:00.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18bdd9b640 1 -- 172.21.15.153:0/2974707000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a000a720 msgr2=0x7f18c009ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:00.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 client.?.objecter ms_handle_connect 0x7f18a000a720 2024-09-10T20:26:00.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18bdd9b640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a000a720 0x7f18c009ff80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18a8007300 tx=0x7f18a80077c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 1 -- 172.21.15.153:0/2974707000 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a80155f0 con 0x7f18a000a720 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 monclient: dump: 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:00.040 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 1 -- 172.21.15.153:0/2974707000 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18a80157b0 con 0x7f18a000a720 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 1 -- 172.21.15.153:0/2974707000 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18a8015ba0 con 0x7f18a000a720 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 monclient: dump: 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.041 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18c4b38640 5 monclient: authenticate success, global_id 4101 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18c4b38640 10 monclient: _renew_subs 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b7fff640 4 set_mon_vals no callback set 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18c4b38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18c4b38640 1 -- 172.21.15.153:0/2974707000 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c012bbd0 con 0x7f18a000a720 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18c4b38640 10 client.4101.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18c4b38640 10 monclient: _renew_subs 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18c4b38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:00.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18c4b38640 1 -- 172.21.15.153:0/2974707000 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c013d390 con 0x7f18a000a720 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18c4b38640 1 librados: init done 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 1 -- 172.21.15.153:0/2974707000 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f18a8016480 con 0x7f18a000a720 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 1 -- 172.21.15.153:0/2974707000 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f18a801f960 con 0x7f18a000a720 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 10 client.4101.objecter ms_dispatch 0x7f18c0000ba0 osd_map(1..1 src has 1..1) 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 3 client.4101.objecter handle_osd_map got epochs [1,1] > 0 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 3 client.4101.objecter handle_osd_map decoding full epoch 1 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.026+0000 7f18b6ffd640 20 client.4101.objecter dump_active .. 0 homeless 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.030+0000 7f18c4b38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.030+0000 7f18c4b38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.030+0000 7f18c4b38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.030+0000 7f18c4b38640 1 -- 172.21.15.153:0/2974707000 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18c013bb80 con 0x7f18a000a720 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.030+0000 7f18b6ffd640 1 -- 172.21.15.153:0/2974707000 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f18a8015d80 con 0x7f18a000a720 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.030+0000 7f18b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:00.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.030+0000 7f18b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:00.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.098+0000 7f18c4b38640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-10T20:26:00.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.098+0000 7f18c4b38640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-10T20:26:00.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.098+0000 7f18c4b38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:00.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.098+0000 7f18c4b38640 1 -- 172.21.15.153:0/2974707000 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f18c0069df0 con 0x7f18a000a720 2024-09-10T20:26:00.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.162+0000 7f18b6ffd640 1 -- 172.21.15.153:0/2974707000 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f18a8016900 con 0x7f18a000a720 2024-09-10T20:26:00.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.162+0000 7f18b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-10T20:26:00.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.162+0000 7f18b6ffd640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-10T20:26:00.170 INFO:teuthology.orchestra.run.smithi153.stderr:adjusted tunables profile to default 2024-09-10T20:26:00.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 10 librados: watch_flush enter 2024-09-10T20:26:00.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 10 librados: watch_flush exit 2024-09-10T20:26:00.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 20 client.4101.objecter shutdown clearing up homeless session... 2024-09-10T20:26:00.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 10 client.4101.objecter successfully canceled tick 2024-09-10T20:26:00.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 10 monclient: shutdown 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 -- 172.21.15.153:0/2974707000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a000a720 msgr2=0x7f18c009ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a000a720 0x7f18c009ff80 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f18a8007300 tx=0x7f18a80077c0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 -- 172.21.15.153:0/2974707000 shutdown_connections 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f18c013db90 0x7f18c00a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f18a000a720 0x7f18c009ff80 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 --2- 172.21.15.153:0/2974707000 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f18a0003720 0x7f18c009fa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 -- 172.21.15.153:0/2974707000 >> 172.21.15.153:0/2974707000 conn(0x7f18c00ab980 msgr2=0x7f18c00aa840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 -- 172.21.15.153:0/2974707000 shutdown_connections 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 -- 172.21.15.153:0/2974707000 wait complete. 2024-09-10T20:26:00.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:00.166+0000 7f18c4b38640 1 librados: shutdown 2024-09-10T20:26:00.194 INFO:tasks.ceph:check_enable_crimson: False 2024-09-10T20:26:00.194 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-10T20:26:00.195 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:26:00.195 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-10T20:26:00.207 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:26:00.207 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-10T20:26:00.261 DEBUG:teuthology.orchestra.run.smithi153:> set -ex 2024-09-10T20:26:00.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-10T20:26:00.313 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:26:00.313 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-10T20:26:00.327 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:26:00.327 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-10T20:26:00.379 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:26:00.379 DEBUG:teuthology.orchestra.run.smithi176:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-10T20:26:00.432 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new faf817a9-b230-4970-b8c5-e5637fd29193 0 2024-09-10T20:26:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient: get_monmap_and_config 2024-09-10T20:26:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient: build_initial_monmap 2024-09-10T20:26:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient: monmap: 2024-09-10T20:26:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:26:00.593443+0000 2024-09-10T20:26:00.594 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:26:00.593443+0000 2024-09-10T20:26:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:00.595 INFO:teuthology.orchestra.run.smithi176.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:00.595 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:00.597 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 1 Processor -- start 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 1 -- start start 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient: init 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f28440a48e0 0x7f28440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient(hunting): picked mon.noname-a con 0x7f28440a48e0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 0x7f28440a5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient(hunting): picked mon.noname-b con 0x7f28440a0b70 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2844074d40 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28440709b0 con 0x7f2844074d40 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f284406f2c0 con 0x7f28440a48e0 2024-09-10T20:26:00.598 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f28440a5740 con 0x7f28440a0b70 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2849cdb640 10 monclient(hunting): authenticate will time out at 925.463151s 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2843fff640 1 --1- >> v1:172.21.15.176:6789/0 conn(0x7f2844074d40 0x7f284413eab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:54928/0 (socket says 172.21.15.176:54928) 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f2843fff640 1 -- 172.21.15.176:0/3333335315 learned_addr learned my addr 172.21.15.176:0/3333335315 (peer_addr_for_me v1:172.21.15.176:0/0) 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f283affd640 1 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 0x7f28440a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f28437fe640 1 --2- 172.21.15.176:0/3333335315 >> v2:172.21.15.153:3300/0 conn(0x7f28440a48e0 0x7f28440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f283affd640 10 monclient(hunting): get_auth_request con 0x7f28440a0b70 auth_method 0 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f283affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:00.599 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f28437fe640 10 monclient(hunting): get_auth_request con 0x7f28440a48e0 auth_method 0 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f28437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f28437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f28437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f283affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f283affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f283affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f28437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f28437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.589+0000 7f28437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 10 monclient(hunting): handle_auth_done global_id 4109 payload 274 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 1 -- 172.21.15.176:0/3333335315 >> v2:172.21.15.153:3300/0 conn(0x7f28440a48e0 msgr2=0x7f28440a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 1 --2- 172.21.15.176:0/3333335315 >> v2:172.21.15.153:3300/0 conn(0x7f28440a48e0 0x7f28440a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 1 -- 172.21.15.176:0/3333335315 >> v1:172.21.15.176:6789/0 conn(0x7f2844074d40 legacy=0x7f284413eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:00.600 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 10 monclient: _finish_hunting 0 2024-09-10T20:26:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 1 monclient: found mon.noname-b 2024-09-10T20:26:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 1 -- 172.21.15.176:0/3333335315 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2844070c90 con 0x7f28440a0b70 2024-09-10T20:26:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f28437fe640 1 --2- 172.21.15.176:0/3333335315 >> v2:172.21.15.153:3300/0 conn(0x7f28440a48e0 0x7f28440a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 1 -- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 msgr2=0x7f28440a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:26:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 1 -- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 msgr2=0x7f28440a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:26:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 1 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 0x7f28440a5200 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0x7f282c0097c0 tx=0x7f282c01c360 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:00.601 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.593+0000 7f283affd640 1 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 0x7f28440a5200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:00.803 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f283affd640 1 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 0x7f28440a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.803 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f283affd640 10 monclient: get_auth_request con 0x7f28440a0b70 auth_method 0 2024-09-10T20:26:00.803 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f283affd640 0 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 0x7f28440a5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-10T20:26:00.803 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f283affd640 1 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 0x7f28440a5200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.803 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:00.803 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 1 -- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 msgr2=0x7f28440a5200 unknown :-1 s=STATE_CLOSED l=0).mark_down 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 1 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28440a0b70 0x7f28440a5200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 10 monclient: _add_conns ranks=[4,0] 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 1 --2- 172.21.15.176:0/3333335315 >> v2:172.21.15.176:3300/0 conn(0x7f2830003510 0x7f2830005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f2830003510 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28300062c0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 1 -- 172.21.15.176:0/3333335315 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f283000a270 con 0x7f28300062c0 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 1 -- 172.21.15.176:0/3333335315 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f283000b470 con 0x7f2830003510 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f2842ffd640 10 monclient(hunting): ms_handle_reset stray mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 1 --2- 172.21.15.176:0/3333335315 >> v2:172.21.15.176:3300/0 conn(0x7f2830003510 0x7f2830005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.804 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient(hunting): get_auth_request con 0x7f2830003510 auth_method 0 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient(hunting): handle_auth_done global_id 4109 payload 293 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 1 -- 172.21.15.176:0/3333335315 >> v1:172.21.15.153:6789/0 conn(0x7f28300062c0 legacy=0x7f2830008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 1 monclient: found mon.noname-c 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:26:00.805 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.797+0000 7f28437fe640 1 -- 172.21.15.176:0/3333335315 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f283000b730 con 0x7f2830003510 2024-09-10T20:26:00.806 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f28437fe640 1 -- 172.21.15.176:0/3333335315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2830003510 msgr2=0x7f2830005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:00.806 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f28437fe640 1 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2830003510 0x7f2830005990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2834009c00 tx=0x7f282c009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1bff4d47bc93ca0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:00.806 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 1 -- 172.21.15.176:0/3333335315 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2834017070 con 0x7f2830003510 2024-09-10T20:26:00.806 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:00.806 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 10 monclient: dump: 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:00.807 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 1 -- 172.21.15.176:0/3333335315 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2834009e10 con 0x7f2830003510 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 1 -- 172.21.15.176:0/3333335315 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2834009880 con 0x7f2830003510 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 5 monclient: authenticate success, global_id 4109 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 4 set_mon_vals no callback set 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2842ffd640 10 monclient: dump: 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.808 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.809 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:00.809 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:00.809 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:00.809 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:00.809 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:00.809 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:00.812 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 10 monclient: shutdown 2024-09-10T20:26:00.812 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:00.812 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 1 -- 172.21.15.176:0/3333335315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2830003510 msgr2=0x7f2830005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:00.812 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 1 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2830003510 0x7f2830005990 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f2834009c00 tx=0x7f282c009190 comp rx=0 tx=0).stop 2024-09-10T20:26:00.812 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2843fff640 1 -- 172.21.15.176:0/3333335315 reap_dead start 2024-09-10T20:26:00.812 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 1 -- 172.21.15.176:0/3333335315 shutdown_connections 2024-09-10T20:26:00.812 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 1 --2- 172.21.15.176:0/3333335315 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2830003510 0x7f2830005990 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 1 -- 172.21.15.176:0/3333335315 >> 172.21.15.176:0/3333335315 conn(0x7f28440a98f0 msgr2=0x7f28440a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 1 -- 172.21.15.176:0/3333335315 shutdown_connections 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.801+0000 7f2849cdb640 1 -- 172.21.15.176:0/3333335315 wait complete. 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient: build_initial_monmap 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient: monmap: 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:00.813 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 librados: starting msgr at 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 librados: starting objecter 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 Processor -- start 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 -- start start 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 librados: setting wanted keys 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 librados: calling monclient init 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient: init 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2830003510 0x7f28441265e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient(hunting): picked mon.noname-c con 0x7f2830003510 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2844128720 0x7f284413f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:00.814 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient(hunting): picked mon.noname-b con 0x7f2844128720 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f284413fad0 0x7f28441d0450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient(hunting): picked mon.noname-a con 0x7f284413fad0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2844142d80 con 0x7f284413fad0 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2844142c00 con 0x7f2844128720 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2844142f00 con 0x7f2830003510 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2849cdb640 10 monclient(hunting): authenticate will time out at 925.678380s 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2844128720 0x7f284413f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2844128720 0x7f284413f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40318/0 (socket says 172.21.15.176:40318) 2024-09-10T20:26:00.815 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 1 -- 172.21.15.176:0/3451851716 learned_addr learned my addr 172.21.15.176:0/3451851716 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient(hunting): get_auth_request con 0x7f2844128720 auth_method 0 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f28437fe640 1 --2- 172.21.15.176:0/3451851716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2830003510 0x7f28441265e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2843fff640 1 --2- 172.21.15.176:0/3451851716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f284413fad0 0x7f28441d0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f28437fe640 10 monclient(hunting): get_auth_request con 0x7f2830003510 auth_method 0 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f28437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f28437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f28437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:00.816 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2843fff640 10 monclient(hunting): get_auth_request con 0x7f284413fad0 auth_method 0 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2843fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2843fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2843fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f28437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f28437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f28437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2843fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2843fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f2843fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 1 -- 172.21.15.176:0/3451851716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2830003510 msgr2=0x7f28441265e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 1 --2- 172.21.15.176:0/3451851716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2830003510 0x7f28441265e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 1 -- 172.21.15.176:0/3451851716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f284413fad0 msgr2=0x7f28441d0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:00.817 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 1 --2- 172.21.15.176:0/3451851716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f284413fad0 0x7f28441d0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.818 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient: _finish_hunting 0 2024-09-10T20:26:00.818 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 1 monclient: found mon.noname-b 2024-09-10T20:26:00.818 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:00.818 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:26:00.818 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.805+0000 7f283affd640 1 -- 172.21.15.176:0/3451851716 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2844127bc0 con 0x7f2844128720 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f283affd640 1 -- 172.21.15.176:0/3451851716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2844128720 msgr2=0x7f284413f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f283affd640 1 --2- 172.21.15.176:0/3451851716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2844128720 0x7f284413f590 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f282c009710 tx=0x7f282c019100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 client.?.objecter ms_handle_connect 0x7f2844128720 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 1 -- 172.21.15.176:0/3451851716 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c004220 con 0x7f2844128720 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 monclient: dump: 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.820 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 monclient: _finish_auth 0 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 monclient: _check_auth_tickets 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 1 -- 172.21.15.176:0/3451851716 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f282c000e50 con 0x7f2844128720 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 1 -- 172.21.15.176:0/3451851716 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f282c0ae650 con 0x7f2844128720 2024-09-10T20:26:00.821 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 5 monclient: authenticate success, global_id 4111 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2841ffb640 4 set_mon_vals no callback set 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 monclient: dump: 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:00.822 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 10 monclient: _renew_subs 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 1 -- 172.21.15.176:0/3451851716 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2844127580 con 0x7f2844128720 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 10 monclient: _renew_subs 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 1 -- 172.21.15.176:0/3451851716 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2844129680 con 0x7f2844128720 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 1 librados: init done 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 1 -- 172.21.15.176:0/3451851716 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f282c0ae7f0 con 0x7f2844128720 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 1 -- 172.21.15.176:0/3451851716 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f282c0b8750 con 0x7f2844128720 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 10 client.4111.objecter ms_dispatch 0x7f2844000ba0 osd_map(2..2 src has 1..2) 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-09-10T20:26:00.823 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2840ff9640 20 client.4111.objecter dump_active .. 0 homeless 2024-09-10T20:26:00.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:00.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:00.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:00.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.809+0000 7f2849cdb640 1 -- 172.21.15.176:0/3451851716 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28440a4d40 con 0x7f2844128720 2024-09-10T20:26:00.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.817+0000 7f2840ff9640 1 -- 172.21.15.176:0/3451851716 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f282c0bd030 con 0x7f2844128720 2024-09-10T20:26:00.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.817+0000 7f2840ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:00.824 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.817+0000 7f2840ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:00.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.885+0000 7f2849cdb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "faf817a9-b230-4970-b8c5-e5637fd29193", "id": 0}] 2024-09-10T20:26:00.889 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.885+0000 7f2849cdb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "faf817a9-b230-4970-b8c5-e5637fd29193", "id": 0}] 2024-09-10T20:26:00.890 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.885+0000 7f2849cdb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:00.890 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.885+0000 7f2849cdb640 1 -- 172.21.15.176:0/3451851716 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "faf817a9-b230-4970-b8c5-e5637fd29193", "id": 0} v 0) -- 0x7f28440a0060 con 0x7f2844128720 2024-09-10T20:26:00.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.889+0000 7f2840ff9640 1 -- 172.21.15.176:0/3451851716 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "faf817a9-b230-4970-b8c5-e5637fd29193", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f282c016070 con 0x7f2844128720 2024-09-10T20:26:00.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2840ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "faf817a9-b230-4970-b8c5-e5637fd29193", "id": 0}] 2024-09-10T20:26:00.896 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2840ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:00.897 INFO:teuthology.orchestra.run.smithi176.stdout:0 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 10 librados: watch_flush enter 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 10 librados: watch_flush exit 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 20 client.4111.objecter shutdown clearing up homeless session... 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 10 client.4111.objecter successfully canceled tick 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 10 monclient: shutdown 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 -- 172.21.15.176:0/3451851716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2844128720 msgr2=0x7f284413f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 --2- 172.21.15.176:0/3451851716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2844128720 0x7f284413f590 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f282c009710 tx=0x7f282c019100 comp rx=0 tx=0).stop 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 -- 172.21.15.176:0/3451851716 shutdown_connections 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 --2- 172.21.15.176:0/3451851716 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f284413fad0 0x7f28441d0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.899 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 --2- 172.21.15.176:0/3451851716 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2844128720 0x7f284413f590 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.900 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 --2- 172.21.15.176:0/3451851716 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2830003510 0x7f28441265e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:00.900 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 -- 172.21.15.176:0/3451851716 >> 172.21.15.176:0/3451851716 conn(0x7f28440a98f0 msgr2=0x7f2844073380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:00.900 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 -- 172.21.15.176:0/3451851716 shutdown_connections 2024-09-10T20:26:00.900 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 -- 172.21.15.176:0/3451851716 wait complete. 2024-09-10T20:26:00.900 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:00.893+0000 7f2849cdb640 1 librados: shutdown 2024-09-10T20:26:00.913 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new f85962f8-8bd9-4e26-8d66-49b6528617a3 1 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient: get_monmap_and_config 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient: monmap: 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:26:01.029052+0000 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:26:01.029052+0000 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:01.030 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:01.031 INFO:teuthology.orchestra.run.smithi176.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:01.031 INFO:teuthology.orchestra.run.smithi176.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:01.031 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 1 Processor -- start 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 1 -- start start 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient: init 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a37d0 0x7f16480a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient(hunting): picked mon.noname-b con 0x7f16480a37d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16480a40f0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1648138da0 0x7f1648140960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.033 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient(hunting): picked mon.noname-c con 0x7f1648138da0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f164809cbf0 con 0x7f16480a40f0 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f164809de30 con 0x7f1648138da0 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f164809ce60 con 0x7f16480a37d0 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164efc3640 10 monclient(hunting): authenticate will time out at 925.898667s 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1648138da0 0x7f1648140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1648138da0 0x7f1648140960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40320/0 (socket says 172.21.15.176:40320) 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 1 -- 172.21.15.176:0/2914741286 learned_addr learned my addr 172.21.15.176:0/2914741286 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 10 monclient(hunting): get_auth_request con 0x7f1648138da0 auth_method 0 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.034 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164d560640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 1 --2- 172.21.15.176:0/2914741286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a37d0 0x7f16480a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient(hunting): get_auth_request con 0x7f16480a37d0 auth_method 0 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.035 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient(hunting): handle_auth_done global_id 4115 payload 274 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 1 -- 172.21.15.176:0/2914741286 >> v2:172.21.15.176:3300/0 conn(0x7f1648138da0 msgr2=0x7f1648140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 1 --2- 172.21.15.176:0/2914741286 >> v2:172.21.15.176:3300/0 conn(0x7f1648138da0 0x7f1648140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 1 -- 172.21.15.176:0/2914741286 >> v1:172.21.15.176:6789/0 conn(0x7f16480a40f0 legacy=0x7f1648138690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 1 monclient: found mon.noname-b 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:01.036 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.025+0000 7f164cd5f640 1 -- 172.21.15.176:0/2914741286 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164809d140 con 0x7f16480a37d0 2024-09-10T20:26:01.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 -- 172.21.15.176:0/2914741286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a37d0 msgr2=0x7f16480a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 --2- 172.21.15.176:0/2914741286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a37d0 0x7f16480a3bb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f16380097c0 tx=0x7f163801c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2da2edc390c4771f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 1 -- 172.21.15.176:0/2914741286 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16380b0070 con 0x7f16480a37d0 2024-09-10T20:26:01.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:01.038 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 10 monclient: dump: 2024-09-10T20:26:01.038 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.038 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.038 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 1 -- 172.21.15.176:0/2914741286 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16380ab600 con 0x7f16480a37d0 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 1 -- 172.21.15.176:0/2914741286 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16380ab9d0 con 0x7f16480a37d0 2024-09-10T20:26:01.039 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 5 monclient: authenticate success, global_id 4115 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 4 set_mon_vals no callback set 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f16477fe640 10 monclient: dump: 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.040 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient: shutdown 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 -- 172.21.15.176:0/2914741286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a37d0 msgr2=0x7f16480a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 --2- 172.21.15.176:0/2914741286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a37d0 0x7f16480a3bb0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f16380097c0 tx=0x7f163801c360 comp rx=0 tx=0).stop 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 -- 172.21.15.176:0/2914741286 shutdown_connections 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 --2- 172.21.15.176:0/2914741286 >> v2:172.21.15.176:3300/0 conn(0x7f1648138da0 0x7f1648140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 --2- 172.21.15.176:0/2914741286 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a37d0 0x7f16480a3bb0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 -- 172.21.15.176:0/2914741286 >> 172.21.15.176:0/2914741286 conn(0x7f16480a86a0 msgr2=0x7f16480a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 -- 172.21.15.176:0/2914741286 shutdown_connections 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 -- 172.21.15.176:0/2914741286 wait complete. 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient: monmap: 2024-09-10T20:26:01.041 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 librados: starting msgr at 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 librados: starting objecter 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 Processor -- start 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 -- start start 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 librados: setting wanted keys 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 librados: calling monclient init 2024-09-10T20:26:01.042 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient: init 2024-09-10T20:26:01.046 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.046 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:26:01.046 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16480a37d0 0x7f1648120990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.046 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient(hunting): picked mon.noname-b con 0x7f16480a37d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:01.046 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a40f0 0x7f164811e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient(hunting): picked mon.noname-c con 0x7f16480a40f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1648138da0 0x7f1648137800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient(hunting): picked mon.noname-a con 0x7f1648138da0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f16480772a0 con 0x7f1648138da0 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1648077120 con 0x7f16480a37d0 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f16481436c0 con 0x7f16480a40f0 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164efc3640 10 monclient(hunting): authenticate will time out at 925.906500s 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16480a37d0 0x7f1648120990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.047 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16480a37d0 0x7f1648120990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40334/0 (socket says 172.21.15.176:40334) 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 -- 172.21.15.176:0/2649905136 learned_addr learned my addr 172.21.15.176:0/2649905136 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient(hunting): get_auth_request con 0x7f16480a37d0 auth_method 0 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 -- 172.21.15.176:0/2649905136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a40f0 msgr2=0x7f164811e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 --2- 172.21.15.176:0/2649905136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a40f0 0x7f164811e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 -- 172.21.15.176:0/2649905136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1648138da0 msgr2=0x7f1648137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.048 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 --2- 172.21.15.176:0/2649905136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1648138da0 0x7f1648137800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 monclient: found mon.noname-b 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.033+0000 7f164cd5f640 1 -- 172.21.15.176:0/2649905136 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f164809f470 con 0x7f16480a37d0 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164cd5f640 1 -- 172.21.15.176:0/2649905136 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16480a37d0 msgr2=0x7f1648120990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164cd5f640 1 --2- 172.21.15.176:0/2649905136 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16480a37d0 0x7f1648120990 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f163801cc00 tx=0x7f1638019100 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 client.?.objecter ms_handle_connect 0x7f16480a37d0 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 1 -- 172.21.15.176:0/2649905136 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16380b9020 con 0x7f16480a37d0 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:01.049 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 monclient: dump: 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.050 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 1 -- 172.21.15.176:0/2649905136 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16380b5690 con 0x7f16480a37d0 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 5 monclient: authenticate success, global_id 4117 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 10 monclient: _renew_subs 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 1 -- 172.21.15.176:0/2649905136 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f164809f0b0 con 0x7f16480a37d0 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 1 -- 172.21.15.176:0/2649905136 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16380b0040 con 0x7f16480a37d0 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 monclient: dump: 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.051 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16467fc640 4 set_mon_vals no callback set 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 10 monclient: _renew_subs 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 1 -- 172.21.15.176:0/2649905136 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1648122190 con 0x7f16480a37d0 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 1 librados: init done 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 1 -- 172.21.15.176:0/2649905136 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f16380bc020 con 0x7f16480a37d0 2024-09-10T20:26:01.052 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 1 -- 172.21.15.176:0/2649905136 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f16380039d0 con 0x7f16480a37d0 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 10 client.4117.objecter ms_dispatch 0x7f1648000ba0 osd_map(3..3 src has 1..3) 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f16457fa640 20 client.4117.objecter dump_active .. 0 homeless 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.037+0000 7f164efc3640 1 -- 172.21.15.176:0/2649905136 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16480a3bb0 con 0x7f16480a37d0 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.041+0000 7f16457fa640 1 -- 172.21.15.176:0/2649905136 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f16380c3070 con 0x7f16480a37d0 2024-09-10T20:26:01.053 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.041+0000 7f16457fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.054 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.041+0000 7f16457fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:01.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.109+0000 7f164efc3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f85962f8-8bd9-4e26-8d66-49b6528617a3", "id": 1}] 2024-09-10T20:26:01.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.109+0000 7f164efc3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f85962f8-8bd9-4e26-8d66-49b6528617a3", "id": 1}] 2024-09-10T20:26:01.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.109+0000 7f164efc3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.116 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.109+0000 7f164efc3640 1 -- 172.21.15.176:0/2649905136 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f85962f8-8bd9-4e26-8d66-49b6528617a3", "id": 1} v 0) -- 0x7f16480a7ad0 con 0x7f16480a37d0 2024-09-10T20:26:01.121 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f16457fa640 1 -- 172.21.15.176:0/2649905136 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f85962f8-8bd9-4e26-8d66-49b6528617a3", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f1638016070 con 0x7f16480a37d0 2024-09-10T20:26:01.121 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f16457fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f85962f8-8bd9-4e26-8d66-49b6528617a3", "id": 1}] 2024-09-10T20:26:01.121 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f16457fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:01.122 INFO:teuthology.orchestra.run.smithi176.stdout:1 2024-09-10T20:26:01.123 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 10 librados: watch_flush enter 2024-09-10T20:26:01.123 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 10 librados: watch_flush exit 2024-09-10T20:26:01.123 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 20 client.4117.objecter shutdown clearing up homeless session... 2024-09-10T20:26:01.123 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 10 client.4117.objecter successfully canceled tick 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 10 monclient: shutdown 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 -- 172.21.15.176:0/2649905136 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16480a37d0 msgr2=0x7f1648120990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 --2- 172.21.15.176:0/2649905136 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16480a37d0 0x7f1648120990 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f163801cc00 tx=0x7f1638019100 comp rx=0 tx=0).stop 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 -- 172.21.15.176:0/2649905136 shutdown_connections 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 --2- 172.21.15.176:0/2649905136 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1648138da0 0x7f1648137800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 --2- 172.21.15.176:0/2649905136 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16480a40f0 0x7f164811e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 --2- 172.21.15.176:0/2649905136 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16480a37d0 0x7f1648120990 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 -- 172.21.15.176:0/2649905136 >> 172.21.15.176:0/2649905136 conn(0x7f16480a86a0 msgr2=0x7f16480a1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 -- 172.21.15.176:0/2649905136 shutdown_connections 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 -- 172.21.15.176:0/2649905136 wait complete. 2024-09-10T20:26:01.124 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.117+0000 7f164efc3640 1 librados: shutdown 2024-09-10T20:26:01.138 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 18d54b55-1393-42b9-9fcc-84202659cf31 2 2024-09-10T20:26:01.244 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient: get_monmap_and_config 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient: monmap: 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:26:01.243972+0000 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:26:01.243972+0000 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:01.245 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:01.246 INFO:teuthology.orchestra.run.smithi176.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:01.246 INFO:teuthology.orchestra.run.smithi176.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:01.246 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 1 Processor -- start 2024-09-10T20:26:01.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 1 -- start start 2024-09-10T20:26:01.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient: init 2024-09-10T20:26:01.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-10T20:26:01.248 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3aa40a03d0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3aa40a0ec0 0x7f3aa40a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient(hunting): picked mon.noname-c con 0x7f3aa40a0ec0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa413b760 0x7f3aa413db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient(hunting): picked mon.noname-b con 0x7f3aa413b760 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3aa40a2b90 con 0x7f3aa40a03d0 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3aa40a25d0 con 0x7f3aa40a0ec0 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa40a28b0 con 0x7f3aa413b760 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa91e5640 10 monclient(hunting): authenticate will time out at 926.113718s 2024-09-10T20:26:01.249 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3aa40a0ec0 0x7f3aa40a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3aa40a0ec0 0x7f3aa40a2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40348/0 (socket says 172.21.15.176:40348) 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 1 -- 172.21.15.176:0/1647784999 learned_addr learned my addr 172.21.15.176:0/1647784999 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 10 monclient(hunting): get_auth_request con 0x7f3aa40a0ec0 auth_method 0 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa2d9d640 1 --1- 172.21.15.176:0/1647784999 >> v1:172.21.15.176:6789/0 conn(0x7f3aa40a03d0 0x7f3aa40a07b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.176:6789/0 says I am v1:172.21.15.176:54936/0 (socket says 172.21.15.176:54936) 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 1 --2- 172.21.15.176:0/1647784999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa413b760 0x7f3aa413db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient(hunting): get_auth_request con 0x7f3aa413b760 auth_method 0 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.250 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient(hunting): handle_auth_done global_id 4118 payload 274 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 1 -- 172.21.15.176:0/1647784999 >> v2:172.21.15.176:3300/0 conn(0x7f3aa40a0ec0 msgr2=0x7f3aa40a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 1 --2- 172.21.15.176:0/1647784999 >> v2:172.21.15.176:3300/0 conn(0x7f3aa40a0ec0 0x7f3aa40a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 1 -- 172.21.15.176:0/1647784999 >> v1:172.21.15.176:6789/0 conn(0x7f3aa40a03d0 legacy=0x7f3aa40a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 1 monclient: found mon.noname-b 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:01.251 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.241+0000 7f3aa359e640 1 -- 172.21.15.176:0/1647784999 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa40a4b40 con 0x7f3aa413b760 2024-09-10T20:26:01.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa359e640 1 -- 172.21.15.176:0/1647784999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa413b760 msgr2=0x7f3aa413db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa359e640 1 --2- 172.21.15.176:0/1647784999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa413b760 0x7f3aa413db50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009d90 tx=0x7f3a9c01c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2f59526064fd3ac server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.253 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 1 -- 172.21.15.176:0/1647784999 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0a6040 con 0x7f3aa413b760 2024-09-10T20:26:01.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:01.254 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 10 monclient: dump: 2024-09-10T20:26:01.254 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.254 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.254 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.254 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.254 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.254 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 1 -- 172.21.15.176:0/1647784999 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9c0af070 con 0x7f3aa413b760 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 1 -- 172.21.15.176:0/1647784999 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0aa860 con 0x7f3aa413b760 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.255 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa1d9b640 10 monclient: dump: 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa91e5640 5 monclient: authenticate success, global_id 4118 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa91e5640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:01.256 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.245+0000 7f3aa91e5640 4 set_mon_vals no callback set 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient: shutdown 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 -- 172.21.15.176:0/1647784999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa413b760 msgr2=0x7f3aa413db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 --2- 172.21.15.176:0/1647784999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa413b760 0x7f3aa413db50 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c009d90 tx=0x7f3a9c01c650 comp rx=0 tx=0).stop 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 -- 172.21.15.176:0/1647784999 shutdown_connections 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 --2- 172.21.15.176:0/1647784999 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa413b760 0x7f3aa413db50 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 --2- 172.21.15.176:0/1647784999 >> v2:172.21.15.176:3300/0 conn(0x7f3aa40a0ec0 0x7f3aa40a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 -- 172.21.15.176:0/1647784999 >> 172.21.15.176:0/1647784999 conn(0x7f3aa40a9ac0 msgr2=0x7f3aa40a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 -- 172.21.15.176:0/1647784999 shutdown_connections 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 -- 172.21.15.176:0/1647784999 wait complete. 2024-09-10T20:26:01.257 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient: monmap: 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.258 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 librados: starting msgr at 2024-09-10T20:26:01.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 librados: starting objecter 2024-09-10T20:26:01.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 Processor -- start 2024-09-10T20:26:01.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 -- start start 2024-09-10T20:26:01.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 librados: setting wanted keys 2024-09-10T20:26:01.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 librados: calling monclient init 2024-09-10T20:26:01.259 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient: init 2024-09-10T20:26:01.262 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.262 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:26:01.262 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa40a03d0 0x7f3aa406f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.262 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient(hunting): picked mon.noname-c con 0x7f3aa40a03d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:01.262 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3aa40a0ec0 0x7f3aa406f710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.262 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient(hunting): picked mon.noname-a con 0x7f3aa40a0ec0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3aa413b760 0x7f3aa41d0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient(hunting): picked mon.noname-b con 0x7f3aa413b760 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa4140c40 con 0x7f3aa40a0ec0 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3aa4140ac0 con 0x7f3aa413b760 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3aa4140dc0 con 0x7f3aa40a03d0 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa91e5640 10 monclient(hunting): authenticate will time out at 926.121833s 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3aa413b760 0x7f3aa41d0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3aa413b760 0x7f3aa41d0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40364/0 (socket says 172.21.15.176:40364) 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 1 -- 172.21.15.176:0/3219213267 learned_addr learned my addr 172.21.15.176:0/3219213267 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.263 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient(hunting): get_auth_request con 0x7f3aa413b760 auth_method 0 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa2d9d640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa40a03d0 0x7f3aa406f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa259c640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3aa40a0ec0 0x7f3aa406f710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 1 -- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa40a03d0 msgr2=0x7f3aa406f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa40a03d0 0x7f3aa406f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 1 -- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3aa40a0ec0 msgr2=0x7f3aa406f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3aa40a0ec0 0x7f3aa406f710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.264 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 1 monclient: found mon.noname-b 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa359e640 1 -- 172.21.15.176:0/3219213267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3aa41382b0 con 0x7f3aa413b760 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa2d9d640 10 monclient: get_auth_request con 0x7f3aa40a03d0 auth_method 0 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa259c640 10 monclient: get_auth_request con 0x7f3aa40a0ec0 auth_method 0 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa2d9d640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa40a03d0 0x7f3aa406f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.249+0000 7f3aa259c640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3aa40a0ec0 0x7f3aa406f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa359e640 1 -- 172.21.15.176:0/3219213267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3aa413b760 msgr2=0x7f3aa41d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa359e640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3aa413b760 0x7f3aa41d0240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c01cef0 tx=0x7f3a9c0193f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 client.?.objecter ms_handle_connect 0x7f3aa413b760 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:01.265 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 1 -- 172.21.15.176:0/3219213267 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0c2070 con 0x7f3aa413b760 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: dump: 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.266 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 1 -- 172.21.15.176:0/3219213267 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a9c0aab30 con 0x7f3aa413b760 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 5 monclient: authenticate success, global_id 4123 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 10 monclient: _renew_subs 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 1 -- 172.21.15.176:0/3219213267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3aa4138590 con 0x7f3aa413b760 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 1 -- 172.21.15.176:0/3219213267 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0bd420 con 0x7f3aa413b760 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: dump: 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.267 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 10 monclient: _renew_subs 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa159a640 4 set_mon_vals no callback set 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 1 -- 172.21.15.176:0/3219213267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3aa40a00c0 con 0x7f3aa413b760 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 1 librados: init done 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 1 -- 172.21.15.176:0/3219213267 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3a9c0bd640 con 0x7f3aa413b760 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 1 -- 172.21.15.176:0/3219213267 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7f3a9c0045d0 con 0x7f3aa413b760 2024-09-10T20:26:01.268 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 client.4123.objecter ms_dispatch 0x7f3aa4000ba0 osd_map(4..4 src has 1..4) 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 20 client.4123.objecter dump_active .. 0 homeless 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3aa91e5640 1 -- 172.21.15.176:0/3219213267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3aa40a2fb0 con 0x7f3aa413b760 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 1 -- 172.21.15.176:0/3219213267 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f3a9c0b8070 con 0x7f3aa413b760 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.269 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.253+0000 7f3a8b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:01.331 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.325+0000 7f3aa91e5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "18d54b55-1393-42b9-9fcc-84202659cf31", "id": 2}] 2024-09-10T20:26:01.331 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.325+0000 7f3aa91e5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "18d54b55-1393-42b9-9fcc-84202659cf31", "id": 2}] 2024-09-10T20:26:01.331 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.325+0000 7f3aa91e5640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.331 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.325+0000 7f3aa91e5640 1 -- 172.21.15.176:0/3219213267 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "18d54b55-1393-42b9-9fcc-84202659cf31", "id": 2} v 0) -- 0x7f3aa4137e10 con 0x7f3aa413b760 2024-09-10T20:26:01.337 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3a8b7fe640 1 -- 172.21.15.176:0/3219213267 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "18d54b55-1393-42b9-9fcc-84202659cf31", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7f3aa40a2b90 con 0x7f3aa413b760 2024-09-10T20:26:01.337 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3a8b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "18d54b55-1393-42b9-9fcc-84202659cf31", "id": 2}] 2024-09-10T20:26:01.337 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3a8b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:01.337 INFO:teuthology.orchestra.run.smithi176.stdout:2 2024-09-10T20:26:01.338 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 10 librados: watch_flush enter 2024-09-10T20:26:01.338 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 10 librados: watch_flush exit 2024-09-10T20:26:01.338 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 20 client.4123.objecter shutdown clearing up homeless session... 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 10 client.4123.objecter successfully canceled tick 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 10 monclient: shutdown 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 -- 172.21.15.176:0/3219213267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3aa413b760 msgr2=0x7f3aa41d0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3aa413b760 0x7f3aa41d0240 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c01cef0 tx=0x7f3a9c0193f0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 -- 172.21.15.176:0/3219213267 shutdown_connections 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3aa413b760 0x7f3aa41d0240 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3aa40a0ec0 0x7f3aa406f710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 --2- 172.21.15.176:0/3219213267 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3aa40a03d0 0x7f3aa406f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 -- 172.21.15.176:0/3219213267 >> 172.21.15.176:0/3219213267 conn(0x7f3aa40a9ac0 msgr2=0x7f3aa40a84c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 -- 172.21.15.176:0/3219213267 shutdown_connections 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 -- 172.21.15.176:0/3219213267 wait complete. 2024-09-10T20:26:01.339 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.333+0000 7f3aa91e5640 1 librados: shutdown 2024-09-10T20:26:01.353 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 5149b9c4-35d7-4ffb-9fac-7b3d9b660101 3 2024-09-10T20:26:01.470 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient: get_monmap_and_config 2024-09-10T20:26:01.470 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient: monmap: 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:26:01.469721+0000 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:26:01.469721+0000 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:01.471 INFO:teuthology.orchestra.run.smithi176.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:01.472 INFO:teuthology.orchestra.run.smithi176.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:01.472 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.474 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 1 Processor -- start 2024-09-10T20:26:01.474 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 1 -- start start 2024-09-10T20:26:01.474 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient: init 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient: _add_conns ranks=[3,4,1,0,2] 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1f4409a440 0x7f1f4409a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1f4409a440 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1f440a1210 0x7f1f440a2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient(hunting): picked mon.noname-c con 0x7f1f440a1210 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f440a1620 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1f4409feb0 con 0x7f1f440a1620 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f1f4409ee10 con 0x7f1f4409a440 2024-09-10T20:26:01.475 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f1f440a0190 con 0x7f1f440a1210 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f490fb640 10 monclient(hunting): authenticate will time out at 926.339613s 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1f440a1210 0x7f1f440a2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f1f440a1210 0x7f1f440a2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40370/0 (socket says 172.21.15.176:40370) 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 1 -- 172.21.15.176:0/1052205706 learned_addr learned my addr 172.21.15.176:0/1052205706 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 10 monclient(hunting): get_auth_request con 0x7f1f440a1210 auth_method 0 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.476 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.465+0000 7f1f42d9d640 1 --2- 172.21.15.176:0/1052205706 >> v2:172.21.15.153:3300/0 conn(0x7f1f4409a440 0x7f1f4409a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient(hunting): get_auth_request con 0x7f1f4409a440 auth_method 0 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.477 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2024-09-10T20:26:01.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 1 -- 172.21.15.176:0/1052205706 >> v2:172.21.15.176:3300/0 conn(0x7f1f440a1210 msgr2=0x7f1f440a2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 1 --2- 172.21.15.176:0/1052205706 >> v2:172.21.15.176:3300/0 conn(0x7f1f440a1210 0x7f1f440a2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 1 -- 172.21.15.176:0/1052205706 >> v1:172.21.15.176:6789/0 conn(0x7f1f440a1620 legacy=0x7f1f440a3490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 1 monclient: found mon.noname-a 2024-09-10T20:26:01.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:01.478 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.469+0000 7f1f42d9d640 1 -- 172.21.15.176:0/1052205706 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f440a0470 con 0x7f1f4409a440 2024-09-10T20:26:01.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f42d9d640 1 -- 172.21.15.176:0/1052205706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 msgr2=0x7f1f4409a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f42d9d640 1 --2- 172.21.15.176:0/1052205706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 0x7f1f4409a820 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1f44070190 tx=0x7f1f3401c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cf37cead74fb276 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 1 -- 172.21.15.176:0/1052205706 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f340b0070 con 0x7f1f4409a440 2024-09-10T20:26:01.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:01.482 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 10 monclient: dump: 2024-09-10T20:26:01.482 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.483 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 1 -- 172.21.15.176:0/1052205706 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f340ab600 con 0x7f1f4409a440 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 1 -- 172.21.15.176:0/1052205706 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f340ab9d0 con 0x7f1f4409a440 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f41d9b640 10 monclient: dump: 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.484 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.485 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.485 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.485 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.485 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 5 monclient: authenticate success, global_id 4113 2024-09-10T20:26:01.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:01.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 4 set_mon_vals no callback set 2024-09-10T20:26:01.485 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 10 monclient: shutdown 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 -- 172.21.15.176:0/1052205706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 msgr2=0x7f1f4409a820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 --2- 172.21.15.176:0/1052205706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 0x7f1f4409a820 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1f44070190 tx=0x7f1f3401c3d0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 -- 172.21.15.176:0/1052205706 shutdown_connections 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 --2- 172.21.15.176:0/1052205706 >> v2:172.21.15.176:3300/0 conn(0x7f1f440a1210 0x7f1f440a2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 --2- 172.21.15.176:0/1052205706 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 0x7f1f4409a820 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 -- 172.21.15.176:0/1052205706 >> 172.21.15.176:0/1052205706 conn(0x7f1f440a4040 msgr2=0x7f1f440a4420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 -- 172.21.15.176:0/1052205706 shutdown_connections 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 -- 172.21.15.176:0/1052205706 wait complete. 2024-09-10T20:26:01.486 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 10 monclient: monmap: 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.487 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 librados: starting msgr at 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 librados: starting objecter 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 Processor -- start 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 -- start start 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 librados: setting wanted keys 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 librados: calling monclient init 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 10 monclient: init 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 0x7f1f44139970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.473+0000 7f1f490fb640 10 monclient(hunting): picked mon.noname-a con 0x7f1f4409a440 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f440a1210 0x7f1f44139eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.488 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient(hunting): picked mon.noname-b con 0x7f1f440a1210 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f440a1620 0x7f1f4413a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient(hunting): picked mon.noname-c con 0x7f1f440a1620 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1f44068ae0 con 0x7f1f4409a440 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1f44068960 con 0x7f1f440a1210 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1f44068c60 con 0x7f1f440a1620 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient(hunting): authenticate will time out at 926.348077s 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f440a1210 0x7f1f44139eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.489 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f440a1210 0x7f1f44139eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40382/0 (socket says 172.21.15.176:40382) 2024-09-10T20:26:01.490 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 -- 172.21.15.176:0/1973769179 learned_addr learned my addr 172.21.15.176:0/1973769179 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient(hunting): get_auth_request con 0x7f1f440a1210 auth_method 0 2024-09-10T20:26:01.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f42d9d640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 0x7f1f44139970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.491 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f42d9d640 10 monclient(hunting): get_auth_request con 0x7f1f4409a440 auth_method 0 2024-09-10T20:26:01.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f42d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f42d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f42d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4359e640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f440a1620 0x7f1f4413a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient(hunting): handle_auth_done global_id 4129 payload 824 2024-09-10T20:26:01.492 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 -- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f440a1620 msgr2=0x7f1f4413a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f440a1620 0x7f1f4413a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 -- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 msgr2=0x7f1f44139970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 0x7f1f44139970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.493 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.494 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 monclient: found mon.noname-b 2024-09-10T20:26:01.494 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.494 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.494 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 -- 172.21.15.176:0/1973769179 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f4412d210 con 0x7f1f440a1210 2024-09-10T20:26:01.494 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4359e640 10 monclient: get_auth_request con 0x7f1f440a1620 auth_method 0 2024-09-10T20:26:01.494 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4359e640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f440a1620 0x7f1f4413a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:01.494 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f42d9d640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 0x7f1f44139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:26:01.496 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 -- 172.21.15.176:0/1973769179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f440a1210 msgr2=0x7f1f44139eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.496 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 client.?.objecter ms_handle_connect 0x7f1f440a1210 2024-09-10T20:26:01.496 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4259c640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f440a1210 0x7f1f44139eb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00e890 tx=0x7f1f2c00ed50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.496 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:01.496 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 1 -- 172.21.15.176:0/1973769179 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c017070 con 0x7f1f440a1210 2024-09-10T20:26:01.496 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 monclient: dump: 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.497 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 1 -- 172.21.15.176:0/1973769179 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1f2c00cb50 con 0x7f1f440a1210 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 1 -- 172.21.15.176:0/1973769179 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2c013470 con 0x7f1f440a1210 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f3b7fe640 10 monclient: dump: 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.498 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f4159a640 4 set_mon_vals no callback set 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 5 monclient: authenticate success, global_id 4129 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient: _renew_subs 2024-09-10T20:26:01.499 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 1 -- 172.21.15.176:0/1973769179 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f4409b450 con 0x7f1f440a1210 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient: _renew_subs 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 1 -- 172.21.15.176:0/1973769179 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f4412cf30 con 0x7f1f440a1210 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.477+0000 7f1f490fb640 1 librados: init done 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f3b7fe640 1 -- 172.21.15.176:0/1973769179 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1f2c01a070 con 0x7f1f440a1210 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f3b7fe640 1 -- 172.21.15.176:0/1973769179 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f1f2c013be0 con 0x7f1f440a1210 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f3b7fe640 10 client.4129.objecter ms_dispatch 0x7f1f44000ba0 osd_map(5..5 src has 1..5) 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f3b7fe640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f3b7fe640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-09-10T20:26:01.500 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f3b7fe640 20 client.4129.objecter dump_active .. 0 homeless 2024-09-10T20:26:01.501 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f490fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.501 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f490fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.501 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f490fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.501 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f490fb640 1 -- 172.21.15.176:0/1973769179 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f4409a820 con 0x7f1f440a1210 2024-09-10T20:26:01.501 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f3b7fe640 1 -- 172.21.15.176:0/1973769179 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f1f2c013610 con 0x7f1f440a1210 2024-09-10T20:26:01.501 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f3b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.501 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.481+0000 7f1f3b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:01.553 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.549+0000 7f1f490fb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "5149b9c4-35d7-4ffb-9fac-7b3d9b660101", "id": 3}] 2024-09-10T20:26:01.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.549+0000 7f1f490fb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "5149b9c4-35d7-4ffb-9fac-7b3d9b660101", "id": 3}] 2024-09-10T20:26:01.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.549+0000 7f1f490fb640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.554 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.549+0000 7f1f490fb640 1 -- 172.21.15.176:0/1973769179 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "5149b9c4-35d7-4ffb-9fac-7b3d9b660101", "id": 3} v 0) -- 0x7f1f44069da0 con 0x7f1f440a1210 2024-09-10T20:26:01.559 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.553+0000 7f1f3b7fe640 1 -- 172.21.15.176:0/1973769179 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "5149b9c4-35d7-4ffb-9fac-7b3d9b660101", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f1f2c0059a0 con 0x7f1f440a1210 2024-09-10T20:26:01.560 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.553+0000 7f1f3b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "5149b9c4-35d7-4ffb-9fac-7b3d9b660101", "id": 3}] 2024-09-10T20:26:01.560 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.553+0000 7f1f3b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:01.560 INFO:teuthology.orchestra.run.smithi176.stdout:3 2024-09-10T20:26:01.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 10 librados: watch_flush enter 2024-09-10T20:26:01.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 10 librados: watch_flush exit 2024-09-10T20:26:01.561 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 20 client.4129.objecter shutdown clearing up homeless session... 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 10 client.4129.objecter successfully canceled tick 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 10 monclient: shutdown 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 -- 172.21.15.176:0/1973769179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f440a1210 msgr2=0x7f1f44139eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f440a1210 0x7f1f44139eb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1f2c00e890 tx=0x7f1f2c00ed50 comp rx=0 tx=0).stop 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 -- 172.21.15.176:0/1973769179 shutdown_connections 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1f440a1620 0x7f1f4413a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1f440a1210 0x7f1f44139eb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 --2- 172.21.15.176:0/1973769179 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1f4409a440 0x7f1f44139970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 -- 172.21.15.176:0/1973769179 >> 172.21.15.176:0/1973769179 conn(0x7f1f440a4040 msgr2=0x7f1f440a5410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 -- 172.21.15.176:0/1973769179 shutdown_connections 2024-09-10T20:26:01.562 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 -- 172.21.15.176:0/1973769179 wait complete. 2024-09-10T20:26:01.563 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.557+0000 7f1f490fb640 1 librados: shutdown 2024-09-10T20:26:01.577 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 61e0c0cc-4301-4947-9f1d-838d2d7bbfea 4 2024-09-10T20:26:01.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient: get_monmap_and_config 2024-09-10T20:26:01.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.694 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient: monmap: 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:26:01.693540+0000 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:26:01.693540+0000 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:01.695 INFO:teuthology.orchestra.run.smithi176.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:01.696 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 1 Processor -- start 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 1 -- start start 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient: init 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbd400a03d0 0x7fbd400a2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient(hunting): picked mon.noname-a con 0x7fbd400a03d0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a07b0 0x7fbd400a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient(hunting): picked mon.noname-b con 0x7fbd400a07b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:01.699 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd4012c5f0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd400a2b90 con 0x7fbd4012c5f0 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fbd400a25d0 con 0x7fbd400a03d0 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbd400a28b0 con 0x7fbd400a07b0 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.689+0000 7fbd44cd3640 10 monclient(hunting): authenticate will time out at 926.563709s 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a07b0 0x7fbd400a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3e59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbd400a03d0 0x7fbd400a2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.700 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a07b0 0x7fbd400a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.176:33986/0 (socket says 172.21.15.176:33986) 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3ed9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fbd4012c5f0 0x7fbd4012ca80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.176:54828/0 (socket says 172.21.15.176:54828) 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/3025830012 learned_addr learned my addr 172.21.15.176:0/3025830012 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient(hunting): get_auth_request con 0x7fbd400a07b0 auth_method 0 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3e59c640 10 monclient(hunting): get_auth_request con 0x7fbd400a03d0 auth_method 0 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.701 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient(hunting): handle_auth_done global_id 4121 payload 274 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/3025830012 >> v2:172.21.15.153:3300/0 conn(0x7fbd400a03d0 msgr2=0x7fbd400a2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 1 --2- 172.21.15.176:0/3025830012 >> v2:172.21.15.153:3300/0 conn(0x7fbd400a03d0 0x7fbd400a2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/3025830012 >> v1:172.21.15.153:6789/0 conn(0x7fbd4012c5f0 legacy=0x7fbd4012ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 1 monclient: found mon.noname-b 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.702 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:01.703 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/3025830012 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd400a1110 con 0x7fbd400a07b0 2024-09-10T20:26:01.703 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.693+0000 7fbd3e59c640 1 --2- 172.21.15.176:0/3025830012 >> v2:172.21.15.153:3300/0 conn(0x7fbd400a03d0 0x7fbd400a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:01.708 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/3025830012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a07b0 msgr2=0x7fbd400a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.708 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 --2- 172.21.15.176:0/3025830012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a07b0 0x7fbd400a0bd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009870 tx=0x7fbd3401c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f7eccca3d739677 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 1 -- 172.21.15.176:0/3025830012 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd340b0070 con 0x7fbd400a07b0 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 10 monclient: dump: 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.709 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.710 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.710 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.710 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 5 monclient: authenticate success, global_id 4121 2024-09-10T20:26:01.710 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:01.711 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 1 -- 172.21.15.176:0/3025830012 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd340ab600 con 0x7fbd400a07b0 2024-09-10T20:26:01.711 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.711 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:01.711 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 4 set_mon_vals no callback set 2024-09-10T20:26:01.711 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 1 -- 172.21.15.176:0/3025830012 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd340ab9d0 con 0x7fbd400a07b0 2024-09-10T20:26:01.711 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.711 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:01.711 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3d59a640 10 monclient: dump: 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient: shutdown 2024-09-10T20:26:01.712 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 -- 172.21.15.176:0/3025830012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a07b0 msgr2=0x7fbd400a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 --2- 172.21.15.176:0/3025830012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a07b0 0x7fbd400a0bd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fbd34009870 tx=0x7fbd3401c2d0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 -- 172.21.15.176:0/3025830012 shutdown_connections 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 --2- 172.21.15.176:0/3025830012 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a07b0 0x7fbd400a0bd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 --2- 172.21.15.176:0/3025830012 >> v2:172.21.15.153:3300/0 conn(0x7fbd400a03d0 0x7fbd400a2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 -- 172.21.15.176:0/3025830012 >> 172.21.15.176:0/3025830012 conn(0x7fbd400a9ac0 msgr2=0x7fbd400a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 -- 172.21.15.176:0/3025830012 shutdown_connections 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 -- 172.21.15.176:0/3025830012 wait complete. 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient: monmap: 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.713 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:01.714 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-09-10T20:26:01.714 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-09-10T20:26:01.714 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.714 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.714 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:01.714 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:01.714 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:01.714 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 librados: starting msgr at 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 librados: starting objecter 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 Processor -- start 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 -- start start 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 librados: setting wanted keys 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 librados: calling monclient init 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient: init 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a03d0 0x7fbd4012fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient(hunting): picked mon.noname-c con 0x7fbd400a03d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:01.715 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd400a07b0 0x7fbd4009d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient(hunting): picked mon.noname-b con 0x7fbd400a07b0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd4012c5f0 0x7fbd4009d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient(hunting): picked mon.noname-a con 0x7fbd4012c5f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd40141330 con 0x7fbd4012c5f0 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbd401411b0 con 0x7fbd400a07b0 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbd401414b0 con 0x7fbd400a03d0 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.716 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd44cd3640 10 monclient(hunting): authenticate will time out at 926.571376s 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd400a07b0 0x7fbd4009d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd400a07b0 0x7fbd4009d1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40396/0 (socket says 172.21.15.176:40396) 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/1794059387 learned_addr learned my addr 172.21.15.176:0/1794059387 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient(hunting): get_auth_request con 0x7fbd400a07b0 auth_method 0 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3e59c640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a03d0 0x7fbd4012fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.717 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3ed9d640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd4012c5f0 0x7fbd4009d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a03d0 msgr2=0x7fbd4012fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a03d0 0x7fbd4012fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd4012c5f0 msgr2=0x7fbd4009d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd4012c5f0 0x7fbd4009d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 monclient: found mon.noname-b 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/1794059387 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd40130ba0 con 0x7fbd400a07b0 2024-09-10T20:26:01.718 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3e59c640 10 monclient: get_auth_request con 0x7fbd400a03d0 auth_method 0 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3ed9d640 10 monclient: get_auth_request con 0x7fbd4012c5f0 auth_method 0 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3e59c640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a03d0 0x7fbd4012fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.697+0000 7fbd3ed9d640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd4012c5f0 0x7fbd4009d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd3dd9b640 1 -- 172.21.15.176:0/1794059387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd400a07b0 msgr2=0x7fbd4009d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd3dd9b640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd400a07b0 0x7fbd4009d1c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd340ae040 tx=0x7fbd34019070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 client.?.objecter ms_handle_connect 0x7fbd400a07b0 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 1 -- 172.21.15.176:0/1794059387 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd340b0040 con 0x7fbd400a07b0 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 monclient: dump: 2024-09-10T20:26:01.719 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.720 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 1 -- 172.21.15.176:0/1794059387 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd340b4620 con 0x7fbd400a07b0 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 5 monclient: authenticate success, global_id 4132 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 10 monclient: _renew_subs 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 1 -- 172.21.15.176:0/1794059387 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd40135c00 con 0x7fbd400a07b0 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 1 -- 172.21.15.176:0/1794059387 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd340b4a10 con 0x7fbd400a07b0 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 monclient: dump: 2024-09-10T20:26:01.721 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.722 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.722 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.722 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.722 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.722 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.722 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.722 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.722 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.722 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.723 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd3cd99640 4 set_mon_vals no callback set 2024-09-10T20:26:01.723 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 10 client.4132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:01.723 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 10 monclient: _renew_subs 2024-09-10T20:26:01.723 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.723 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 1 -- 172.21.15.176:0/1794059387 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd401303d0 con 0x7fbd400a07b0 2024-09-10T20:26:01.723 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 1 librados: init done 2024-09-10T20:26:01.723 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 1 -- 172.21.15.176:0/1794059387 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fbd340b4bb0 con 0x7fbd400a07b0 2024-09-10T20:26:01.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 1 -- 172.21.15.176:0/1794059387 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7fbd340aa3a0 con 0x7fbd400a07b0 2024-09-10T20:26:01.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 10 client.4132.objecter ms_dispatch 0x7fbd40000ba0 osd_map(6..6 src has 1..6) 2024-09-10T20:26:01.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 3 client.4132.objecter handle_osd_map got epochs [6,6] > 0 2024-09-10T20:26:01.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 3 client.4132.objecter handle_osd_map decoding full epoch 6 2024-09-10T20:26:01.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd2effd640 20 client.4132.objecter dump_active .. 0 homeless 2024-09-10T20:26:01.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.724 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.701+0000 7fbd44cd3640 1 -- 172.21.15.176:0/1794059387 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd400a2fb0 con 0x7fbd400a07b0 2024-09-10T20:26:01.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.705+0000 7fbd2effd640 1 -- 172.21.15.176:0/1794059387 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fbd340abe20 con 0x7fbd400a07b0 2024-09-10T20:26:01.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.705+0000 7fbd2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.725 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.705+0000 7fbd2effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:01.780 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.777+0000 7fbd44cd3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "61e0c0cc-4301-4947-9f1d-838d2d7bbfea", "id": 4}] 2024-09-10T20:26:01.781 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.777+0000 7fbd44cd3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "61e0c0cc-4301-4947-9f1d-838d2d7bbfea", "id": 4}] 2024-09-10T20:26:01.781 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.777+0000 7fbd44cd3640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.781 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.777+0000 7fbd44cd3640 1 -- 172.21.15.176:0/1794059387 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "61e0c0cc-4301-4947-9f1d-838d2d7bbfea", "id": 4} v 0) -- 0x7fbd4009e320 con 0x7fbd400a07b0 2024-09-10T20:26:01.787 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd2effd640 1 -- 172.21.15.176:0/1794059387 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "61e0c0cc-4301-4947-9f1d-838d2d7bbfea", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7fbd400a2b90 con 0x7fbd400a07b0 2024-09-10T20:26:01.787 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "61e0c0cc-4301-4947-9f1d-838d2d7bbfea", "id": 4}] 2024-09-10T20:26:01.787 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd2effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:01.787 INFO:teuthology.orchestra.run.smithi176.stdout:4 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 10 librados: watch_flush enter 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 10 librados: watch_flush exit 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 20 client.4132.objecter shutdown clearing up homeless session... 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 10 client.4132.objecter successfully canceled tick 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 10 monclient: shutdown 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 -- 172.21.15.176:0/1794059387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd400a07b0 msgr2=0x7fbd4009d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd400a07b0 0x7fbd4009d1c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbd340ae040 tx=0x7fbd34019070 comp rx=0 tx=0).stop 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 -- 172.21.15.176:0/1794059387 shutdown_connections 2024-09-10T20:26:01.789 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd4012c5f0 0x7fbd4009d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.790 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd400a07b0 0x7fbd4009d1c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.790 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 --2- 172.21.15.176:0/1794059387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd400a03d0 0x7fbd4012fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.790 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 -- 172.21.15.176:0/1794059387 >> 172.21.15.176:0/1794059387 conn(0x7fbd400a9ac0 msgr2=0x7fbd401348b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:01.790 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 -- 172.21.15.176:0/1794059387 shutdown_connections 2024-09-10T20:26:01.790 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 -- 172.21.15.176:0/1794059387 wait complete. 2024-09-10T20:26:01.790 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.781+0000 7fbd44cd3640 1 librados: shutdown 2024-09-10T20:26:01.803 DEBUG:teuthology.orchestra.run.smithi176:> sudo ceph --cluster ceph osd new 2feacd82-c45a-4d90-8210-ff9faecc2d79 5 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient: get_monmap_and_config 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient: monmap: 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:26:01.919926+0000 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:26:01.919926+0000 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.921 INFO:teuthology.orchestra.run.smithi176.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:01.922 INFO:teuthology.orchestra.run.smithi176.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:01.922 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:01.922 INFO:teuthology.orchestra.run.smithi176.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:01.922 INFO:teuthology.orchestra.run.smithi176.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:01.922 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.925 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 1 Processor -- start 2024-09-10T20:26:01.925 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 1 -- start start 2024-09-10T20:26:01.925 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient: init 2024-09-10T20:26:01.925 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.925 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient: _add_conns ranks=[0,2,4,1,3] 2024-09-10T20:26:01.925 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39040a5a40 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:01.925 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 0x7f39040a3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.925 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient(hunting): picked mon.noname-b con 0x7f39040a6530 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f390413db90 0x7f390413ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient(hunting): picked mon.noname-c con 0x7f390413db90 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39040a6b70 con 0x7f39040a5a40 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f39040708a0 con 0x7f390413db90 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f39040a26b0 con 0x7f39040a6530 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f390949f640 10 monclient(hunting): authenticate will time out at 926.789719s 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f390413db90 0x7f390413ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.926 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f390413db90 0x7f390413ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40402/0 (socket says 172.21.15.176:40402) 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 1 -- 172.21.15.176:0/1007628812 learned_addr learned my addr 172.21.15.176:0/1007628812 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient(hunting): get_auth_request con 0x7f390413db90 auth_method 0 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39027fc640 1 --2- 172.21.15.176:0/1007628812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 0x7f39040a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient(hunting): handle_auth_done global_id 4135 payload 274 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 1 -- 172.21.15.176:0/1007628812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 msgr2=0x7f39040a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.927 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 1 --2- 172.21.15.176:0/1007628812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 0x7f39040a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 1 -- 172.21.15.176:0/1007628812 >> v1:172.21.15.153:6789/0 conn(0x7f39040a5a40 legacy=0x7f39040a5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:01.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 1 monclient: found mon.noname-c 2024-09-10T20:26:01.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39037fe640 1 -- 172.21.15.176:0/1007628812 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39040705f0 con 0x7f390413db90 2024-09-10T20:26:01.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39027fc640 10 monclient: get_auth_request con 0x7f39040a6530 auth_method 0 2024-09-10T20:26:01.928 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.917+0000 7f39027fc640 1 --2- 172.21.15.176:0/1007628812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 0x7f39040a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:01.934 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f39037fe640 1 -- 172.21.15.176:0/1007628812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 msgr2=0x7f390413ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.934 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f39037fe640 1 --2- 172.21.15.176:0/1007628812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 0x7f390413ff80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3904070a20 tx=0x7f38fc01c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e9b7a24288ffb8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.934 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 1 -- 172.21.15.176:0/1007628812 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0a5510 con 0x7f390413db90 2024-09-10T20:26:01.934 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.934 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:01.934 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 10 monclient: dump: 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.935 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 1 -- 172.21.15.176:0/1007628812 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38fc0a56b0 con 0x7f390413db90 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 5 monclient: authenticate success, global_id 4135 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 1 -- 172.21.15.176:0/1007628812 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0a5a80 con 0x7f390413db90 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 4 set_mon_vals no callback set 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:01.936 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f3901ffb640 10 monclient: dump: 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 10 monclient: shutdown 2024-09-10T20:26:01.937 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 1 -- 172.21.15.176:0/1007628812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 msgr2=0x7f390413ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 1 --2- 172.21.15.176:0/1007628812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 0x7f390413ff80 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3904070a20 tx=0x7f38fc01c8d0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 1 -- 172.21.15.176:0/1007628812 shutdown_connections 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 1 --2- 172.21.15.176:0/1007628812 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 0x7f390413ff80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 1 --2- 172.21.15.176:0/1007628812 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 0x7f39040a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 1 -- 172.21.15.176:0/1007628812 >> 172.21.15.176:0/1007628812 conn(0x7f39040ab980 msgr2=0x7f39040a7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 1 -- 172.21.15.176:0/1007628812 shutdown_connections 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.921+0000 7f390949f640 1 -- 172.21.15.176:0/1007628812 wait complete. 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient: build_initial_monmap 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient: monmap: 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 0 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:01.938 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 0.000000 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:created 0.000000 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 librados: starting msgr at 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 librados: starting objecter 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 Processor -- start 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 -- start start 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 librados: setting wanted keys 2024-09-10T20:26:01.939 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 librados: calling monclient init 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient: init 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39040a5a40 0x7f3904137fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient(hunting): picked mon.noname-a con 0x7f39040a5a40 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 0x7f39041384e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient(hunting): picked mon.noname-c con 0x7f39040a6530 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 0x7f390409f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient(hunting): picked mon.noname-b con 0x7f390413db90 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3904143070 con 0x7f39040a5a40 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.940 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3904142ef0 con 0x7f390413db90 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f39041431f0 con 0x7f39040a6530 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient(hunting): authenticate will time out at 926.797094s 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 0x7f390409f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 0x7f390409f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:40406/0 (socket says 172.21.15.176:40406) 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 -- 172.21.15.176:0/1311310707 learned_addr learned my addr 172.21.15.176:0/1311310707 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient(hunting): get_auth_request con 0x7f390413db90 auth_method 0 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.941 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f3902ffd640 1 --2- 172.21.15.176:0/1311310707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39040a5a40 0x7f3904137fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f3902ffd640 10 monclient(hunting): get_auth_request con 0x7f39040a5a40 auth_method 0 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39027fc640 1 --2- 172.21.15.176:0/1311310707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 0x7f39041384e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f3902ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f3902ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f3902ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient(hunting): handle_auth_done global_id 4138 payload 824 2024-09-10T20:26:01.942 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 -- 172.21.15.176:0/1311310707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 msgr2=0x7f39041384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 --2- 172.21.15.176:0/1311310707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 0x7f39041384e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 -- 172.21.15.176:0/1311310707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39040a5a40 msgr2=0x7f3904137fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 --2- 172.21.15.176:0/1311310707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39040a5a40 0x7f3904137fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 monclient: found mon.noname-b 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 -- 172.21.15.176:0/1311310707 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39040aaaf0 con 0x7f390413db90 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 -- 172.21.15.176:0/1311310707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 msgr2=0x7f390409f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39037fe640 1 --2- 172.21.15.176:0/1311310707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 0x7f390409f920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f38fc01d170 tx=0x7f38fc019670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 10 client.?.objecter ms_handle_connect 0x7f390413db90 2024-09-10T20:26:01.943 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 1 -- 172.21.15.176:0/1311310707 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0bc020 con 0x7f390413db90 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 10 monclient: dump: 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.944 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 1 -- 172.21.15.176:0/1311310707 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f38fc0b8a90 con 0x7f390413db90 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f38fb7fe640 1 -- 172.21.15.176:0/1311310707 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0a5510 con 0x7f390413db90 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 5 monclient: authenticate success, global_id 4138 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f390949f640 10 monclient: _renew_subs 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.925+0000 7f39017fa640 4 set_mon_vals no callback set 2024-09-10T20:26:01.945 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 1 -- 172.21.15.176:0/1311310707 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3904122a60 con 0x7f390413db90 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 10 monclient: dump: 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:epoch 1 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:election_strategy: 1 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:01.946 INFO:teuthology.orchestra.run.smithi176.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 10 monclient: _renew_subs 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 1 -- 172.21.15.176:0/1311310707 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39040a2230 con 0x7f390413db90 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 1 librados: init done 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 1 -- 172.21.15.176:0/1311310707 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f38fc0bf020 con 0x7f390413db90 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 1 -- 172.21.15.176:0/1311310707 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f38fc0c1d10 con 0x7f390413db90 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 10 client.4138.objecter ms_dispatch 0x7f3904000ba0 osd_map(7..7 src has 1..7) 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 3 client.4138.objecter handle_osd_map got epochs [7,7] > 0 2024-09-10T20:26:01.947 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 3 client.4138.objecter handle_osd_map decoding full epoch 7 2024-09-10T20:26:01.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 20 client.4138.objecter dump_active .. 0 homeless 2024-09-10T20:26:01.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:01.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f390949f640 1 -- 172.21.15.176:0/1311310707 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39040a3930 con 0x7f390413db90 2024-09-10T20:26:01.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 1 -- 172.21.15.176:0/1311310707 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f38fc0c6030 con 0x7f390413db90 2024-09-10T20:26:01.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:01.948 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:01.929+0000 7f38fb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:02.005 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.001+0000 7f390949f640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "2feacd82-c45a-4d90-8210-ff9faecc2d79", "id": 5}] 2024-09-10T20:26:02.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.001+0000 7f390949f640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "2feacd82-c45a-4d90-8210-ff9faecc2d79", "id": 5}] 2024-09-10T20:26:02.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.001+0000 7f390949f640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:02.006 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.001+0000 7f390949f640 1 -- 172.21.15.176:0/1311310707 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2feacd82-c45a-4d90-8210-ff9faecc2d79", "id": 5} v 0) -- 0x7f39040a1870 con 0x7f390413db90 2024-09-10T20:26:02.895 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.889+0000 7f38fb7fe640 1 -- 172.21.15.176:0/1311310707 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2feacd82-c45a-4d90-8210-ff9faecc2d79", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f39040aaaf0 con 0x7f390413db90 2024-09-10T20:26:02.895 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.889+0000 7f38fb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "2feacd82-c45a-4d90-8210-ff9faecc2d79", "id": 5}] 2024-09-10T20:26:02.895 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.889+0000 7f38fb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:02.896 INFO:teuthology.orchestra.run.smithi176.stdout:5 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.889+0000 7f390949f640 10 librados: watch_flush enter 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 10 librados: watch_flush exit 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 20 client.4138.objecter shutdown clearing up homeless session... 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 10 client.4138.objecter successfully canceled tick 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 10 monclient: shutdown 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 -- 172.21.15.176:0/1311310707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 msgr2=0x7f390409f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 --2- 172.21.15.176:0/1311310707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 0x7f390409f920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f38fc01d170 tx=0x7f38fc019670 comp rx=0 tx=0).stop 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 -- 172.21.15.176:0/1311310707 shutdown_connections 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 --2- 172.21.15.176:0/1311310707 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f390413db90 0x7f390409f920 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 --2- 172.21.15.176:0/1311310707 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39040a6530 0x7f39041384e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:02.897 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 --2- 172.21.15.176:0/1311310707 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39040a5a40 0x7f3904137fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:02.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 -- 172.21.15.176:0/1311310707 >> 172.21.15.176:0/1311310707 conn(0x7f39040ab980 msgr2=0x7f39040a4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:02.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 -- 172.21.15.176:0/1311310707 shutdown_connections 2024-09-10T20:26:02.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 -- 172.21.15.176:0/1311310707 wait complete. 2024-09-10T20:26:02.898 INFO:teuthology.orchestra.run.smithi176.stderr:2024-09-10T20:26:02.893+0000 7f390949f640 1 librados: shutdown 2024-09-10T20:26:02.917 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-10T20:26:02.917 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-10T20:26:02.920 INFO:tasks.ceph.osd.0:Started 2024-09-10T20:26:02.920 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-10T20:26:02.920 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-10T20:26:02.922 INFO:tasks.ceph.osd.1:Started 2024-09-10T20:26:02.923 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-10T20:26:02.923 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-10T20:26:02.925 INFO:tasks.ceph.osd.2:Started 2024-09-10T20:26:02.925 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-10T20:26:02.925 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-10T20:26:02.928 INFO:tasks.ceph.osd.3:Started 2024-09-10T20:26:02.928 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-10T20:26:02.928 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-10T20:26:02.930 INFO:tasks.ceph.osd.4:Started 2024-09-10T20:26:02.930 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-10T20:26:02.931 DEBUG:teuthology.orchestra.run.smithi176:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-10T20:26:02.933 INFO:tasks.ceph.osd.5:Started 2024-09-10T20:26:02.933 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:26:03.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.046+0000 7f239e32f640 10 monclient: get_monmap_and_config 2024-09-10T20:26:03.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.046+0000 7f239e32f640 10 monclient: build_initial_monmap 2024-09-10T20:26:03.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.046+0000 7f239e32f640 10 monclient: monmap: 2024-09-10T20:26:03.053 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:03.053 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:03.052663+0000 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:03.052663+0000 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:03.054 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.046+0000 7f239e32f640 1 Processor -- start 2024-09-10T20:26:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.046+0000 7f239e32f640 1 -- start start 2024-09-10T20:26:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.046+0000 7f239e32f640 10 monclient: init 2024-09-10T20:26:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.046+0000 7f239e32f640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.046+0000 7f239e32f640 10 monclient: _add_conns ranks=[0,3,1,4,2] 2024-09-10T20:26:03.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.046+0000 7f239e32f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f23980a5ab0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f239e32f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f23980a65a0 0x7f239813dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f239e32f640 10 monclient(hunting): picked mon.noname-a con 0x7f23980a65a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f239e32f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f239813e1b0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f239e32f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2398070780 con 0x7f23980a5ab0 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f239e32f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23980709c0 con 0x7f239813e1b0 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f239e32f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f239e32f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f23980a3010 con 0x7f23980a65a0 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f239e32f640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f239e32f640 10 monclient(hunting): authenticate will time out at 923.192069s 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f2397fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f23980a5ab0 0x7f23980a5e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39408/0 (socket says 172.21.15.153:39408) 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f2397fff640 1 -- 172.21.15.153:0/1395760379 learned_addr learned my addr 172.21.15.153:0/1395760379 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 1 --2- 172.21.15.153:0/1395760379 >> v2:172.21.15.153:3300/0 conn(0x7f23980a65a0 0x7f239813dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:03.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient(hunting): get_auth_request con 0x7f23980a65a0 auth_method 0 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient(hunting): handle_auth_done global_id 4140 payload 274 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 1 -- 172.21.15.153:0/1395760379 >> v1:172.21.15.176:6789/0 conn(0x7f239813e1b0 legacy=0x7f23981405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 1 -- 172.21.15.153:0/1395760379 >> v1:172.21.15.153:6789/0 conn(0x7f23980a5ab0 legacy=0x7f23980a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 1 monclient: found mon.noname-a 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:03.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.050+0000 7f23977fe640 1 -- 172.21.15.153:0/1395760379 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23980a26e0 con 0x7f23980a65a0 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 1 -- 172.21.15.153:0/1395760379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a65a0 msgr2=0x7f239813dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 1 --2- 172.21.15.153:0/1395760379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a65a0 0x7f239813dc70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f23800097c0 tx=0x7f238001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5344c0be40b8aa7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 1 -- 172.21.15.153:0/1395760379 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23800b0070 con 0x7f23980a65a0 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 10 monclient: dump: 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.060 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:03.061 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.061 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:03.061 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:03.061 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:03.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:03.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:03.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:03.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 1 -- 172.21.15.153:0/1395760379 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23800ab600 con 0x7f23980a65a0 2024-09-10T20:26:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 1 -- 172.21.15.153:0/1395760379 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23800ab9d0 con 0x7f23980a65a0 2024-09-10T20:26:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 5 monclient: authenticate success, global_id 4140 2024-09-10T20:26:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 4 set_mon_vals no callback set 2024-09-10T20:26:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:03.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2396ffd640 10 monclient: dump: 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient: shutdown 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 -- 172.21.15.153:0/1395760379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a65a0 msgr2=0x7f239813dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:03.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 --2- 172.21.15.153:0/1395760379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a65a0 0x7f239813dc70 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f23800097c0 tx=0x7f238001c360 comp rx=0 tx=0).stop 2024-09-10T20:26:03.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 -- 172.21.15.153:0/1395760379 shutdown_connections 2024-09-10T20:26:03.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 --2- 172.21.15.153:0/1395760379 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a65a0 0x7f239813dc70 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 -- 172.21.15.153:0/1395760379 >> 172.21.15.153:0/1395760379 conn(0x7f23980aba00 msgr2=0x7f23980a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:03.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 -- 172.21.15.153:0/1395760379 shutdown_connections 2024-09-10T20:26:03.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 -- 172.21.15.153:0/1395760379 wait complete. 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient: build_initial_monmap 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient: monmap: 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:03.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 librados: starting msgr at 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 librados: starting objecter 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 Processor -- start 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 -- start start 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 librados: setting wanted keys 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 librados: calling monclient init 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient: init 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a5ab0 0x7f23980a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient(hunting): picked mon.noname-a con 0x7f23980a5ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23980a65a0 0x7f2398138580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient(hunting): picked mon.noname-c con 0x7f23980a65a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:03.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f239813e1b0 0x7f23981d0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient(hunting): picked mon.noname-b con 0x7f239813e1b0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f23980a3b70 con 0x7f23980a5ab0 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f23981444e0 con 0x7f239813e1b0 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2398122000 con 0x7f23980a65a0 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239e32f640 10 monclient(hunting): authenticate will time out at 923.199156s 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a5ab0 0x7f23980a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a5ab0 0x7f23980a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57244/0 (socket says 172.21.15.153:57244) 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 1 -- 172.21.15.153:0/2174518041 learned_addr learned my addr 172.21.15.153:0/2174518041 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:03.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 10 monclient(hunting): get_auth_request con 0x7f23980a5ab0 auth_method 0 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23980a65a0 0x7f2398138580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient(hunting): get_auth_request con 0x7f23980a65a0 auth_method 0 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:03.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239c8cc640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f239813e1b0 0x7f23981d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient(hunting): handle_auth_done global_id 4133 payload 824 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 1 -- 172.21.15.153:0/2174518041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f239813e1b0 msgr2=0x7f23981d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f239813e1b0 0x7f23981d0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 1 -- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a5ab0 msgr2=0x7f23980a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a5ab0 0x7f23980a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 1 monclient: found mon.noname-c 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f23977fe640 1 -- 172.21.15.153:0/2174518041 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2398141770 con 0x7f23980a65a0 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239c8cc640 10 monclient: get_auth_request con 0x7f239813e1b0 auth_method 0 2024-09-10T20:26:03.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f239c8cc640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f239813e1b0 0x7f23981d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.054+0000 7f2397fff640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a5ab0 0x7f23980a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f23977fe640 1 -- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23980a65a0 msgr2=0x7f2398138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f23977fe640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23980a65a0 0x7f2398138580 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23800ad000 tx=0x7f2380019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 client.?.objecter ms_handle_connect 0x7f23980a65a0 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 1 -- 172.21.15.153:0/2174518041 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23800b9070 con 0x7f23980a65a0 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 monclient: dump: 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:03.076 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 monclient: _finish_auth 0 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 monclient: _check_auth_tickets 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 5 monclient: authenticate success, global_id 4133 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 10 monclient: _renew_subs 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 1 -- 172.21.15.153:0/2174518041 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23800b5890 con 0x7f23980a65a0 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 1 -- 172.21.15.153:0/2174518041 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23980a9040 con 0x7f23980a65a0 2024-09-10T20:26:03.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 1 -- 172.21.15.153:0/2174518041 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23800b0040 con 0x7f23980a65a0 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 monclient: dump: 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:03.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 10 client.4133.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 10 monclient: _renew_subs 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 1 -- 172.21.15.153:0/2174518041 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2398121d20 con 0x7f23980a65a0 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f23967fc640 4 set_mon_vals no callback set 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 1 librados: init done 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 1 -- 172.21.15.153:0/2174518041 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f23800bc070 con 0x7f23980a65a0 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 1 -- 172.21.15.153:0/2174518041 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f23800bed60 con 0x7f23980a65a0 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 10 client.4133.objecter ms_dispatch 0x7f2398000ba0 osd_map(8..8 src has 1..8) 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 3 client.4133.objecter handle_osd_map got epochs [8,8] > 0 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 3 client.4133.objecter handle_osd_map decoding full epoch 8 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f2394ff9640 20 client.4133.objecter dump_active .. 0 homeless 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:03.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.058+0000 7f239e32f640 1 -- 172.21.15.153:0/2174518041 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23980a5e90 con 0x7f23980a65a0 2024-09-10T20:26:03.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.062+0000 7f2394ff9640 1 -- 172.21.15.153:0/2174518041 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7f2380016070 con 0x7f23980a65a0 2024-09-10T20:26:03.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.062+0000 7f2394ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:03.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.062+0000 7f2394ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:03.091 INFO:tasks.ceph.mgr.x.smithi176.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-10T20:26:03.091 INFO:tasks.ceph.mgr.x.smithi176.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-10T20:26:03.091 INFO:tasks.ceph.mgr.x.smithi176.stderr: from numpy import show_config as show_numpy_config 2024-09-10T20:26:03.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:03.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:03.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.127 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 -- 172.21.15.153:0/2174518041 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f23980a6be0 con 0x7f23980a65a0 2024-09-10T20:26:03.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f2394ff9640 1 -- 172.21.15.153:0/2174518041 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7f23800b5d40 con 0x7f23980a65a0 2024-09-10T20:26:03.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f2394ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:03.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f2394ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:03.128 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:03.128 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":8,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","created":"2024-09-10T20:25:59.587062+0000","modified":"2024-09-10T20:26:02.414479+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T20:26:02.414479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"faf817a9-b230-4970-b8c5-e5637fd29193","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f85962f8-8bd9-4e26-8d66-49b6528617a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"18d54b55-1393-42b9-9fcc-84202659cf31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5149b9c4-35d7-4ffb-9fac-7b3d9b660101","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61e0c0cc-4301-4947-9f1d-838d2d7bbfea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2feacd82-c45a-4d90-8210-ff9faecc2d79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:26:03.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 10 librados: watch_flush enter 2024-09-10T20:26:03.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 10 librados: watch_flush exit 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 20 client.4133.objecter shutdown clearing up homeless session... 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 10 client.4133.objecter successfully canceled tick 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 10 monclient: shutdown 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 -- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23980a65a0 msgr2=0x7f2398138580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23980a65a0 0x7f2398138580 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23800ad000 tx=0x7f2380019100 comp rx=0 tx=0).stop 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 -- 172.21.15.153:0/2174518041 shutdown_connections 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f239813e1b0 0x7f23981d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f23980a65a0 0x7f2398138580 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 --2- 172.21.15.153:0/2174518041 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f23980a5ab0 0x7f23980a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 -- 172.21.15.153:0/2174518041 >> 172.21.15.153:0/2174518041 conn(0x7f23980aba00 msgr2=0x7f23980aa810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 -- 172.21.15.153:0/2174518041 shutdown_connections 2024-09-10T20:26:03.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 -- 172.21.15.153:0/2174518041 wait complete. 2024-09-10T20:26:03.131 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:03.122+0000 7f239e32f640 1 librados: shutdown 2024-09-10T20:26:03.144 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-10T20:26:03.144 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-10T20:26:03.749 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T20:26:03.865 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.858+0000 7fc2ba35d640 10 monclient: get_monmap_and_config 2024-09-10T20:26:03.865 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.858+0000 7fc2ba35d640 10 monclient: build_initial_monmap 2024-09-10T20:26:03.865 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.858+0000 7fc2ba35d640 10 monclient: monmap: 2024-09-10T20:26:03.865 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:26:03.864697+0000 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:26:03.864697+0000 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:03.866 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:03.868 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.858+0000 7fc2ba35d640 1 Processor -- start 2024-09-10T20:26:03.868 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.858+0000 7fc2ba35d640 1 -- start start 2024-09-10T20:26:03.868 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.858+0000 7fc2ba35d640 10 monclient: init 2024-09-10T20:26:03.868 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:03.868 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-10T20:26:03.868 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc2b4138310 0x7fc2b40a4f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:03.868 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2b4138310 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:03.868 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc2b4138d00 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b41354c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2b40a5480 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2b40a6150 con 0x7fc2b4138d00 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fc2b41386f0 con 0x7fc2b4138310 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b40a5f10 con 0x7fc2b40a5480 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2ba35d640 10 monclient(hunting): authenticate will time out at 924.004365s 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b41354c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc2b4138310 0x7fc2b40a4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:03.869 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b41354c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:39110/0 (socket says 172.21.15.153:39110) 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 1 -- 172.21.15.153:0/3997916158 learned_addr learned my addr 172.21.15.153:0/3997916158 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b3fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc2b4138310 0x7fc2b40a4f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57258/0 (socket says 172.21.15.153:57258) 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient(hunting): get_auth_request con 0x7fc2b40a5480 auth_method 0 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b3fff640 10 monclient(hunting): get_auth_request con 0x7fc2b4138310 auth_method 0 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:03.870 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient(hunting): handle_auth_done global_id 4136 payload 274 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 1 -- 172.21.15.153:0/3997916158 >> v2:172.21.15.153:3300/0 conn(0x7fc2b4138310 msgr2=0x7fc2b40a4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 1 --2- 172.21.15.153:0/3997916158 >> v2:172.21.15.153:3300/0 conn(0x7fc2b4138310 0x7fc2b40a4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 1 -- 172.21.15.153:0/3997916158 >> v1:172.21.15.176:6789/0 conn(0x7fc2b4138d00 legacy=0x7fc2b4134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient: _finish_hunting 0 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 1 monclient: found mon.noname-b 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.871 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.862+0000 7fc2b88fa640 1 -- 172.21.15.153:0/3997916158 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b40a7670 con 0x7fc2b40a5480 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 1 -- 172.21.15.153:0/3997916158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 msgr2=0x7fc2b41354c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 1 --2- 172.21.15.153:0/3997916158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b41354c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009e80 tx=0x7fc2ac01c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a884b794205086d4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 1 -- 172.21.15.153:0/3997916158 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0a5700 con 0x7fc2b40a5480 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 10 monclient: dump: 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.873 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 5 monclient: authenticate success, global_id 4136 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 1 -- 172.21.15.153:0/3997916158 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2b40a67e0 con 0x7fc2b40a5480 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:03.874 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 4 set_mon_vals no callback set 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 1 -- 172.21.15.153:0/3997916158 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac0a5ac0 con 0x7fc2b40a5480 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b2ffd640 10 monclient: dump: 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.875 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient: shutdown 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 -- 172.21.15.153:0/3997916158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 msgr2=0x7fc2b41354c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 --2- 172.21.15.153:0/3997916158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b41354c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac009e80 tx=0x7fc2ac01c8a0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 -- 172.21.15.153:0/3997916158 shutdown_connections 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 --2- 172.21.15.153:0/3997916158 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b41354c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 --2- 172.21.15.153:0/3997916158 >> v2:172.21.15.153:3300/0 conn(0x7fc2b4138310 0x7fc2b40a4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 -- 172.21.15.153:0/3997916158 >> 172.21.15.153:0/3997916158 conn(0x7fc2b40ab240 msgr2=0x7fc2b40ab650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 -- 172.21.15.153:0/3997916158 shutdown_connections 2024-09-10T20:26:03.876 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 -- 172.21.15.153:0/3997916158 wait complete. 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient: build_initial_monmap 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient: monmap: 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 librados: starting msgr at 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 librados: starting objecter 2024-09-10T20:26:03.877 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 Processor -- start 2024-09-10T20:26:03.878 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 -- start start 2024-09-10T20:26:03.878 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 librados: setting wanted keys 2024-09-10T20:26:03.878 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 librados: calling monclient init 2024-09-10T20:26:03.878 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient: init 2024-09-10T20:26:03.881 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:03.881 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:03.881 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b4131bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient(hunting): picked mon.noname-c con 0x7fc2b40a5480 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4138310 0x7fc2b4132130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient(hunting): picked mon.noname-b con 0x7fc2b4138310 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2b4071eb0 0x7fc2b4072340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient(hunting): picked mon.noname-a con 0x7fc2b4071eb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4143050 con 0x7fc2b4071eb0 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b4142ed0 con 0x7fc2b4138310 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b41431d0 con 0x7fc2b40a5480 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2ba35d640 10 monclient(hunting): authenticate will time out at 924.011389s 2024-09-10T20:26:03.882 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b4131bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b4131bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:39120/0 (socket says 172.21.15.153:39120) 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 1 -- 172.21.15.153:0/1399940470 learned_addr learned my addr 172.21.15.153:0/1399940470 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient(hunting): get_auth_request con 0x7fc2b40a5480 auth_method 0 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2b4071eb0 0x7fc2b4072340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 10 monclient(hunting): get_auth_request con 0x7fc2b4071eb0 auth_method 0 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:03.883 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b88fa640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b37fe640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4138310 0x7fc2b4132130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient(hunting): handle_auth_done global_id 4139 payload 824 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 1 -- 172.21.15.153:0/1399940470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4138310 msgr2=0x7fc2b4132130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4138310 0x7fc2b4132130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 1 -- 172.21.15.153:0/1399940470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2b4071eb0 msgr2=0x7fc2b4072340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2b4071eb0 0x7fc2b4072340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 1 monclient: found mon.noname-c 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.884 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b3fff640 1 -- 172.21.15.153:0/1399940470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b40a4c70 con 0x7fc2b40a5480 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b37fe640 10 monclient: get_auth_request con 0x7fc2b4138310 auth_method 0 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.866+0000 7fc2b37fe640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4138310 0x7fc2b4132130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b3fff640 1 -- 172.21.15.153:0/1399940470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 msgr2=0x7fc2b4131bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b3fff640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b4131bf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400e8b0 tx=0x7fc2a400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 client.?.objecter ms_handle_connect 0x7fc2b40a5480 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 1 -- 172.21.15.153:0/1399940470 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4017070 con 0x7fc2b40a5480 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 monclient: dump: 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.885 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 monclient: _finish_auth 0 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 monclient: _check_auth_tickets 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 1 -- 172.21.15.153:0/1399940470 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a400cb50 con 0x7fc2b40a5480 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 1 -- 172.21.15.153:0/1399940470 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a4013470 con 0x7fc2b40a5480 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 5 monclient: authenticate success, global_id 4139 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 10 monclient: _renew_subs 2024-09-10T20:26:03.886 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.888 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 1 -- 172.21.15.153:0/1399940470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b413b190 con 0x7fc2b40a5480 2024-09-10T20:26:03.888 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b1ffb640 4 set_mon_vals no callback set 2024-09-10T20:26:03.888 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:03.888 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:03.888 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 monclient: dump: 2024-09-10T20:26:03.888 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:03.888 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:03.888 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.888 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 10 monclient: _renew_subs 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 1 -- 172.21.15.153:0/1399940470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b413aeb0 con 0x7fc2b40a5480 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 1 librados: init done 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 1 -- 172.21.15.153:0/1399940470 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc2a401a070 con 0x7fc2b40a5480 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 1 -- 172.21.15.153:0/1399940470 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7fc2a4013c50 con 0x7fc2b40a5480 2024-09-10T20:26:03.889 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 10 client.4139.objecter ms_dispatch 0x7fc2b4000ba0 osd_map(8..8 src has 1..8) 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 3 client.4139.objecter handle_osd_map got epochs [8,8] > 0 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 3 client.4139.objecter handle_osd_map decoding full epoch 8 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2b0ff9640 20 client.4139.objecter dump_active .. 0 homeless 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.870+0000 7fc2ba35d640 1 -- 172.21.15.153:0/1399940470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b40a4f40 con 0x7fc2b40a5480 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.874+0000 7fc2b0ff9640 1 -- 172.21.15.153:0/1399940470 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (secure 0 0 0) 0x7fc2b40a4c70 con 0x7fc2b40a5480 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.874+0000 7fc2b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:03.890 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.874+0000 7fc2b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:03.935 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:03.935 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:03.935 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:03.936 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 -- 172.21.15.153:0/1399940470 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2b4074860 con 0x7fc2b40a5480 2024-09-10T20:26:03.936 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2b0ff9640 1 -- 172.21.15.153:0/1399940470 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (secure 0 0 0) 0x7fc2a4013610 con 0x7fc2b40a5480 2024-09-10T20:26:03.936 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:03.936 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:03.937 INFO:teuthology.misc.health.smithi153.stdout: 2024-09-10T20:26:03.937 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":8,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","created":"2024-09-10T20:25:59.587062+0000","modified":"2024-09-10T20:26:02.414479+0000","last_up_change":"0.000000","last_in_change":"2024-09-10T20:26:02.414479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"faf817a9-b230-4970-b8c5-e5637fd29193","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f85962f8-8bd9-4e26-8d66-49b6528617a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"18d54b55-1393-42b9-9fcc-84202659cf31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5149b9c4-35d7-4ffb-9fac-7b3d9b660101","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61e0c0cc-4301-4947-9f1d-838d2d7bbfea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2feacd82-c45a-4d90-8210-ff9faecc2d79","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 10 librados: watch_flush enter 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 10 librados: watch_flush exit 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 20 client.4139.objecter shutdown clearing up homeless session... 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 10 client.4139.objecter successfully canceled tick 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 10 monclient: shutdown 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 -- 172.21.15.153:0/1399940470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 msgr2=0x7fc2b4131bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b4131bf0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc2a400e8b0 tx=0x7fc2a400ed70 comp rx=0 tx=0).stop 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 -- 172.21.15.153:0/1399940470 shutdown_connections 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc2b4071eb0 0x7fc2b4072340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.938 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc2b4138310 0x7fc2b4132130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.939 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 --2- 172.21.15.153:0/1399940470 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc2b40a5480 0x7fc2b4131bf0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:03.939 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 -- 172.21.15.153:0/1399940470 >> 172.21.15.153:0/1399940470 conn(0x7fc2b40ab240 msgr2=0x7fc2b40aa820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:03.939 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 -- 172.21.15.153:0/1399940470 shutdown_connections 2024-09-10T20:26:03.939 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 -- 172.21.15.153:0/1399940470 wait complete. 2024-09-10T20:26:03.939 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:03.930+0000 7fc2ba35d640 1 librados: shutdown 2024-09-10T20:26:03.950 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-10T20:26:04.074 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-09-10T20:26:04.069+0000 7f91e688b7c0 -1 Falling back to public interface 2024-09-10T20:26:04.308 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-09-10T20:26:04.302+0000 7f77930f27c0 -1 Falling back to public interface 2024-09-10T20:26:04.315 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-09-10T20:26:04.310+0000 7fe9927dd7c0 -1 Falling back to public interface 2024-09-10T20:26:04.342 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-09-10T20:26:04.337+0000 7fb9a2c197c0 -1 Falling back to public interface 2024-09-10T20:26:04.614 INFO:tasks.ceph.osd.3.smithi176.stderr:2024-09-10T20:26:04.609+0000 7fb55a4ce7c0 -1 Falling back to public interface 2024-09-10T20:26:04.773 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-09-10T20:26:04.766+0000 7f59d21167c0 -1 Falling back to public interface 2024-09-10T20:26:06.244 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-09-10T20:26:06.237+0000 7f13368a8640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-10T20:26:06.245 INFO:tasks.ceph.mgr.x.smithi176.stderr:2024-09-10T20:26:06.241+0000 7f13368a8640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-10T20:26:08.129 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-09-10T20:26:08.125+0000 7f91e688b7c0 -1 osd.5 0 log_to_monitors true 2024-09-10T20:26:08.335 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-09-10T20:26:08.330+0000 7fe9927dd7c0 -1 osd.1 0 log_to_monitors true 2024-09-10T20:26:08.345 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-09-10T20:26:08.338+0000 7f77930f27c0 -1 osd.0 0 log_to_monitors true 2024-09-10T20:26:08.367 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-09-10T20:26:08.361+0000 7fb9a2c197c0 -1 osd.4 0 log_to_monitors true 2024-09-10T20:26:08.769 INFO:tasks.ceph.osd.3.smithi176.stderr:2024-09-10T20:26:08.765+0000 7fb55a4ce7c0 -1 osd.3 0 log_to_monitors true 2024-09-10T20:26:08.999 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-09-10T20:26:08.994+0000 7f59d21167c0 -1 osd.2 0 log_to_monitors true 2024-09-10T20:26:09.260 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-09-10T20:26:09.253+0000 7f91e2834640 -1 osd.5 0 waiting for initial osdmap 2024-09-10T20:26:09.268 INFO:tasks.ceph.osd.5.smithi176.stderr:2024-09-10T20:26:09.261+0000 7f91de66e640 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:26:10.258 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-09-10T20:26:10.253+0000 7fb99ebc2640 -1 osd.4 0 waiting for initial osdmap 2024-09-10T20:26:10.417 INFO:tasks.ceph.osd.3.smithi176.stderr:2024-09-10T20:26:10.261+0000 7fb556477640 -1 osd.3 0 waiting for initial osdmap 2024-09-10T20:26:10.417 INFO:tasks.ceph.osd.4.smithi176.stderr:2024-09-10T20:26:10.265+0000 7fb99a9fc640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:26:10.417 INFO:tasks.ceph.osd.3.smithi176.stderr:2024-09-10T20:26:10.269+0000 7fb5522b1640 -1 osd.3 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:26:10.618 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.750+0000 7f33dff36640 10 monclient: get_monmap_and_config 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.750+0000 7f33dff36640 10 monclient: build_initial_monmap 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.750+0000 7f33dff36640 10 monclient: monmap: 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:26:10.756384+0000 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:26:10.756384+0000 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:10.757 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:10.758 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:10.758 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:10.758 INFO:teuthology.misc.health.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:10.758 INFO:teuthology.misc.health.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:10.758 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.750+0000 7f33dff36640 1 Processor -- start 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.750+0000 7f33dff36640 1 -- start start 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.750+0000 7f33dff36640 10 monclient: init 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f33d8138310 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f33d8138d00 0x7f33d813d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 10 monclient(hunting): picked mon.noname-a con 0x7f33d8138d00 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f33d813dbd0 0x7f33d813ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:10.760 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 10 monclient(hunting): picked mon.noname-c con 0x7f33d813dbd0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33d80a6150 con 0x7f33d8138310 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f33d81386f0 con 0x7f33d8138d00 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f33d80a5f10 con 0x7f33d813dbd0 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dff36640 10 monclient(hunting): authenticate will time out at 930.896725s 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33ddcd2640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f33d8138310 0x7f33d80a4f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39424/0 (socket says 172.21.15.153:39424) 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f33d8138d00 0x7f33d813d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33ddcd2640 1 -- 172.21.15.153:0/10391035 learned_addr learned my addr 172.21.15.153:0/10391035 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient(hunting): get_auth_request con 0x7f33d8138d00 auth_method 0 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:10.761 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33de4d3640 1 --2- 172.21.15.153:0/10391035 >> v2:172.21.15.176:3300/0 conn(0x7f33d813dbd0 0x7f33d813ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33de4d3640 10 monclient(hunting): get_auth_request con 0x7f33d813dbd0 auth_method 0 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33de4d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33de4d3640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33de4d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33de4d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33de4d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33de4d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:10.762 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient(hunting): handle_auth_done global_id 4179 payload 274 2024-09-10T20:26:10.763 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 1 -- 172.21.15.153:0/10391035 >> v2:172.21.15.176:3300/0 conn(0x7f33d813dbd0 msgr2=0x7f33d813ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:10.763 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 1 --2- 172.21.15.153:0/10391035 >> v2:172.21.15.176:3300/0 conn(0x7f33d813dbd0 0x7f33d813ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:10.763 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 1 -- 172.21.15.153:0/10391035 >> v1:172.21.15.153:6789/0 conn(0x7f33d8138310 legacy=0x7f33d80a4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:10.763 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient: _finish_hunting 0 2024-09-10T20:26:10.763 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 1 monclient: found mon.noname-a 2024-09-10T20:26:10.763 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:10.763 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:10.763 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.754+0000 7f33dd4d1640 1 -- 172.21.15.153:0/10391035 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d80a7670 con 0x7f33d8138d00 2024-09-10T20:26:10.765 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dd4d1640 1 -- 172.21.15.153:0/10391035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138d00 msgr2=0x7f33d813d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:10.765 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dd4d1640 1 --2- 172.21.15.153:0/10391035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138d00 0x7f33d813d690 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33c80097c0 tx=0x7f33c801c2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9901cc01a5ec6712 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 1 -- 172.21.15.153:0/10391035 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c80b0070 con 0x7f33d8138d00 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 10 monclient: dump: 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:10.767 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 10 monclient: _finish_auth 0 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 10 monclient: _check_auth_tickets 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 1 -- 172.21.15.153:0/10391035 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33c80ab600 con 0x7f33d8138d00 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dff36640 5 monclient: authenticate success, global_id 4179 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dff36640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dccd0640 1 -- 172.21.15.153:0/10391035 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33c80ab9d0 con 0x7f33d8138d00 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dff36640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.758+0000 7f33dff36640 4 set_mon_vals no callback set 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 10 monclient: shutdown 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:10.768 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 -- 172.21.15.153:0/10391035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138d00 msgr2=0x7f33d813d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:10.769 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 --2- 172.21.15.153:0/10391035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138d00 0x7f33d813d690 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33c80097c0 tx=0x7f33c801c2f0 comp rx=0 tx=0).stop 2024-09-10T20:26:10.769 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dccd0640 10 monclient: discarding stray monitor message mon_map magic: 0 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 -- 172.21.15.153:0/10391035 shutdown_connections 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 --2- 172.21.15.153:0/10391035 >> v2:172.21.15.176:3300/0 conn(0x7f33d813dbd0 0x7f33d813ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 --2- 172.21.15.153:0/10391035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138d00 0x7f33d813d690 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f33c80097c0 tx=0x7f33c801c2f0 comp rx=0 tx=0).stop 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 -- 172.21.15.153:0/10391035 >> 172.21.15.153:0/10391035 conn(0x7f33d80ab240 msgr2=0x7f33d80ab650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 -- 172.21.15.153:0/10391035 shutdown_connections 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 -- 172.21.15.153:0/10391035 wait complete. 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 10 monclient: build_initial_monmap 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 10 monclient: monmap: 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:10.770 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 librados: starting msgr at 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 librados: starting objecter 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 Processor -- start 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 -- start start 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 librados: setting wanted keys 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 1 librados: calling monclient init 2024-09-10T20:26:10.771 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.762+0000 7f33dff36640 10 monclient: init 2024-09-10T20:26:10.774 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:10.774 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:26:10.774 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138310 0x7f33d813b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:10.774 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33ddcd2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138310 0x7f33d813b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:10.774 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33ddcd2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138310 0x7f33d813b6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57338/0 (socket says 172.21.15.153:57338) 2024-09-10T20:26:10.774 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33ddcd2640 1 -- 172.21.15.153:0/3850570564 learned_addr learned my addr 172.21.15.153:0/3850570564 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:10.774 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient(hunting): picked mon.noname-a con 0x7f33d8138310 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:10.774 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33d813dbd0 0x7f33d813bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient(hunting): picked mon.noname-b con 0x7f33d813dbd0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33d81d02f0 0x7f33d81d0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33d81d02f0 0x7f33d81d0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dd4d1640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33d813dbd0 0x7f33d813bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient(hunting): picked mon.noname-c con 0x7f33d81d02f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f33d8142ea0 con 0x7f33d8138310 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f33d8142d20 con 0x7f33d813dbd0 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f33d8143020 con 0x7f33d81d02f0 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dff36640 10 monclient(hunting): authenticate will time out at 930.909309s 2024-09-10T20:26:10.775 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33ddcd2640 10 monclient(hunting): get_auth_request con 0x7f33d8138310 auth_method 0 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33ddcd2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33ddcd2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33ddcd2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient(hunting): get_auth_request con 0x7f33d81d02f0 auth_method 0 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient(hunting): handle_auth_done global_id 4169 payload 824 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 1 -- 172.21.15.153:0/3850570564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33d813dbd0 msgr2=0x7f33d813bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33d813dbd0 0x7f33d813bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:10.776 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 1 -- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138310 msgr2=0x7f33d813b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:10.777 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138310 0x7f33d813b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:10.777 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient: _finish_hunting 0 2024-09-10T20:26:10.777 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 1 monclient: found mon.noname-c 2024-09-10T20:26:10.777 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:10.777 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:10.777 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33de4d3640 1 -- 172.21.15.153:0/3850570564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33d8137890 con 0x7f33d81d02f0 2024-09-10T20:26:10.777 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dd4d1640 10 monclient: get_auth_request con 0x7f33d813dbd0 auth_method 0 2024-09-10T20:26:10.777 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.766+0000 7f33dd4d1640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33d813dbd0 0x7f33d813bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:10.779 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33de4d3640 1 -- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33d81d02f0 msgr2=0x7f33d81d0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:10.779 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33de4d3640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33d81d02f0 0x7f33d81d0760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f33d000efc0 tx=0x7f33d000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:10.779 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 client.?.objecter ms_handle_connect 0x7f33d81d02f0 2024-09-10T20:26:10.779 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 1 -- 172.21.15.153:0/3850570564 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0018020 con 0x7f33d81d02f0 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 monclient: dump: 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:10.780 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 monclient: _finish_auth 0 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 1 -- 172.21.15.153:0/3850570564 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33d00146a0 con 0x7f33d81d02f0 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33cffff640 4 set_mon_vals no callback set 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 1 -- 172.21.15.153:0/3850570564 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33d0014a90 con 0x7f33d81d02f0 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33ce7fc640 10 monclient: dump: 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:10.781 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33dff36640 5 monclient: authenticate success, global_id 4169 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33dff36640 10 monclient: _renew_subs 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33dff36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.770+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33d80a24a0 con 0x7f33d81d02f0 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33dff36640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33dff36640 10 monclient: _renew_subs 2024-09-10T20:26:10.782 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33dff36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:10.783 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f33d8134a50 con 0x7f33d81d02f0 2024-09-10T20:26:10.783 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33dff36640 1 librados: init done 2024-09-10T20:26:10.783 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33ce7fc640 1 -- 172.21.15.153:0/3850570564 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 4) ==== 48431+0+0 (secure 0 0 0) 0x7f33d001b020 con 0x7f33d81d02f0 2024-09-10T20:26:10.783 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33ce7fc640 1 --2- 172.21.15.153:0/3850570564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f33bc03ce40 0x7f33bc03f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:10.783 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33ce7fc640 1 -- 172.21.15.153:0/3850570564 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(11..11 src has 1..11) ==== 2067+0+0 (secure 0 0 0) 0x7f33d00549f0 con 0x7f33d81d02f0 2024-09-10T20:26:10.783 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33ce7fc640 10 client.4169.objecter ms_dispatch 0x7f33d8000ba0 osd_map(11..11 src has 1..11) 2024-09-10T20:26:10.783 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33ce7fc640 3 client.4169.objecter handle_osd_map got epochs [11,11] > 0 2024-09-10T20:26:10.783 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33ce7fc640 3 client.4169.objecter handle_osd_map decoding full epoch 11 2024-09-10T20:26:10.783 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33ce7fc640 20 client.4169.objecter dump_active .. 0 homeless 2024-09-10T20:26:10.784 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33ddcd2640 1 --2- 172.21.15.153:0/3850570564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f33bc03ce40 0x7f33bc03f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:10.784 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33dff36640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:10.784 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33dff36640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:10.784 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33dff36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:10.784 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33d8133250 con 0x7f33d81d02f0 2024-09-10T20:26:10.784 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.774+0000 7f33ddcd2640 10 monclient: get_auth_request con 0x7f33bc03ce40 auth_method 0 2024-09-10T20:26:10.784 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.778+0000 7f33ce7fc640 10 client.4169.objecter ms_handle_connect 0x7f33bc03ce40 2024-09-10T20:26:10.784 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.778+0000 7f33ddcd2640 1 --2- 172.21.15.153:0/3850570564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f33bc03ce40 0x7f33bc03f300 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f33d4007fd0 tx=0x7f33d4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:10.788 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.782+0000 7f33ce7fc640 1 -- 172.21.15.153:0/3850570564 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f33d0010040 con 0x7f33d81d02f0 2024-09-10T20:26:10.788 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.782+0000 7f33ce7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:10.788 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:10.782+0000 7f33ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:11.045 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.038+0000 7f33dff36640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:11.045 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.038+0000 7f33dff36640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:11.045 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.038+0000 7f33dff36640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:11.045 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.038+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f33d8134270 con 0x7f33d81d02f0 2024-09-10T20:26:11.045 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.038+0000 7f33ce7fc640 1 -- 172.21.15.153:0/3850570564 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+5871 (secure 0 0 0) 0x7f33d0013e20 con 0x7f33d81d02f0 2024-09-10T20:26:11.045 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.038+0000 7f33ce7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:11.045 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.038+0000 7f33ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:11.046 INFO:teuthology.misc.health.smithi153.stdout: 2024-09-10T20:26:11.046 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":11,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","created":"2024-09-10T20:25:59.587062+0000","modified":"2024-09-10T20:26:10.241098+0000","last_up_change":"2024-09-10T20:26:10.241098+0000","last_in_change":"2024-09-10T20:26:02.414479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"faf817a9-b230-4970-b8c5-e5637fd29193","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f85962f8-8bd9-4e26-8d66-49b6528617a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"18d54b55-1393-42b9-9fcc-84202659cf31","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"5149b9c4-35d7-4ffb-9fac-7b3d9b660101","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"61e0c0cc-4301-4947-9f1d-838d2d7bbfea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"2feacd82-c45a-4d90-8210-ff9faecc2d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":2372911183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6801","nonce":2372911183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6803","nonce":2372911183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2372911183}]},"public_addr":"172.21.15.176:6800/2372911183","cluster_addr":"172.21.15.176:6801/2372911183","heartbeat_back_addr":"172.21.15.176:6803/2372911183","heartbeat_front_addr":"172.21.15.176:6802/2372911183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:26:11.049 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 10 librados: watch_flush enter 2024-09-10T20:26:11.049 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 10 librados: watch_flush exit 2024-09-10T20:26:11.049 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 20 client.4169.objecter shutdown clearing up homeless session... 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 10 client.4169.objecter successfully canceled tick 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f33bc03ce40 msgr2=0x7f33bc03f300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 --2- 172.21.15.153:0/3850570564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f33bc03ce40 0x7f33bc03f300 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f33d4007fd0 tx=0x7f33d4009040 comp rx=0 tx=0).stop 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 10 monclient: shutdown 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33d81d02f0 msgr2=0x7f33d81d0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33d81d02f0 0x7f33d81d0760 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f33d000efc0 tx=0x7f33d000c490 comp rx=0 tx=0).stop 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 shutdown_connections 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 --2- 172.21.15.153:0/3850570564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f33bc03ce40 0x7f33bc03f300 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33d81d02f0 0x7f33d81d0760 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f33d813dbd0 0x7f33d813bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 --2- 172.21.15.153:0/3850570564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33d8138310 0x7f33d813b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 >> 172.21.15.153:0/3850570564 conn(0x7f33d80ab240 msgr2=0x7f33d80a9140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:11.050 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 shutdown_connections 2024-09-10T20:26:11.051 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 -- 172.21.15.153:0/3850570564 wait complete. 2024-09-10T20:26:11.051 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:11.042+0000 7f33dff36640 1 librados: shutdown 2024-09-10T20:26:11.073 DEBUG:teuthology.misc:1 of 6 OSDs are up 2024-09-10T20:26:11.278 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-09-10T20:26:11.270+0000 7fe98ef98640 -1 osd.1 0 waiting for initial osdmap 2024-09-10T20:26:11.287 INFO:tasks.ceph.osd.1.smithi153.stderr:2024-09-10T20:26:11.282+0000 7fe9865b8640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:26:11.308 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-09-10T20:26:11.302+0000 7f59ce0bf640 -1 osd.2 0 waiting for initial osdmap 2024-09-10T20:26:11.310 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-09-10T20:26:11.302+0000 7f778f09b640 -1 osd.0 0 waiting for initial osdmap 2024-09-10T20:26:11.314 INFO:tasks.ceph.osd.2.smithi153.stderr:2024-09-10T20:26:11.306+0000 7f59c5ef1640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:26:11.314 INFO:tasks.ceph.osd.0.smithi153.stderr:2024-09-10T20:26:11.306+0000 7f7786ecd640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-10T20:26:17.678 DEBUG:teuthology.orchestra.run.smithi153:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-10T20:26:17.798 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.790+0000 7efd09ef9640 10 monclient: get_monmap_and_config 2024-09-10T20:26:17.798 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.790+0000 7efd09ef9640 10 monclient: build_initial_monmap 2024-09-10T20:26:17.798 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.790+0000 7efd09ef9640 10 monclient: monmap: 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:26:17.797775+0000 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:26:17.797775+0000 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:17.799 INFO:teuthology.misc.health.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:17.800 INFO:teuthology.misc.health.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:17.800 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:17.803 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 1 Processor -- start 2024-09-10T20:26:17.803 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 1 -- start start 2024-09-10T20:26:17.803 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient: init 2024-09-10T20:26:17.803 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:17.803 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-10T20:26:17.804 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7efd041331e0 0x7efd041335c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:17.804 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient(hunting): picked mon.noname-c con 0x7efd041331e0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:17.804 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efd04136290 0x7efd041366c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:17.804 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient(hunting): picked mon.noname-a con 0x7efd04136290 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:17.804 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efd04136c00 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:17.804 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd04132d90 con 0x7efd04136c00 2024-09-10T20:26:17.804 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:17.804 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7efd040a2ac0 con 0x7efd04136290 2024-09-10T20:26:17.805 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:17.805 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7efd040a11e0 con 0x7efd041331e0 2024-09-10T20:26:17.805 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:17.805 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd09ef9640 10 monclient(hunting): authenticate will time out at 937.937434s 2024-09-10T20:26:17.805 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efd04136290 0x7efd041366c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:17.805 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7efd04136290 0x7efd041366c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55986/0 (socket says 172.21.15.153:55986) 2024-09-10T20:26:17.805 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 1 -- 172.21.15.153:0/3482438786 learned_addr learned my addr 172.21.15.153:0/3482438786 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:17.805 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient(hunting): get_auth_request con 0x7efd04136290 auth_method 0 2024-09-10T20:26:17.805 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:17.806 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:17.806 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:17.806 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:17.806 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:17.806 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:17.806 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd037fe640 1 --2- 172.21.15.153:0/3482438786 >> v2:172.21.15.176:3300/0 conn(0x7efd041331e0 0x7efd041335c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:17.806 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient(hunting): handle_auth_done global_id 4194 payload 274 2024-09-10T20:26:17.806 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 1 -- 172.21.15.153:0/3482438786 >> v2:172.21.15.176:3300/0 conn(0x7efd041331e0 msgr2=0x7efd041335c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:17.807 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 1 --2- 172.21.15.153:0/3482438786 >> v2:172.21.15.176:3300/0 conn(0x7efd041331e0 0x7efd041335c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:17.807 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 1 -- 172.21.15.153:0/3482438786 >> v1:172.21.15.176:6789/0 conn(0x7efd04136c00 legacy=0x7efd0413d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:17.807 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:26:17.807 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 1 monclient: found mon.noname-a 2024-09-10T20:26:17.807 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:17.807 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:17.807 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd02ffd640 1 -- 172.21.15.153:0/3482438786 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd040a05b0 con 0x7efd04136290 2024-09-10T20:26:17.807 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd037fe640 10 monclient: get_auth_request con 0x7efd041331e0 auth_method 0 2024-09-10T20:26:17.808 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.794+0000 7efd037fe640 1 --2- 172.21.15.153:0/3482438786 >> v2:172.21.15.176:3300/0 conn(0x7efd041331e0 0x7efd041335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd02ffd640 1 -- 172.21.15.153:0/3482438786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 msgr2=0x7efd041366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd02ffd640 1 --2- 172.21.15.153:0/3482438786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 0x7efd041366c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd0409de10 tx=0x7efcf801c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5668782674068 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 1 -- 172.21.15.153:0/3482438786 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf80b0070 con 0x7efd04136290 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 10 monclient: dump: 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:17.815 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 10 monclient: _finish_auth 0 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 10 monclient: _check_auth_tickets 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 1 -- 172.21.15.153:0/3482438786 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf80ab600 con 0x7efd04136290 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 5 monclient: authenticate success, global_id 4194 2024-09-10T20:26:17.816 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:17.817 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:17.817 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:17.817 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 1 -- 172.21.15.153:0/3482438786 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf80ab9d0 con 0x7efd04136290 2024-09-10T20:26:17.817 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:17.817 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 4 set_mon_vals no callback set 2024-09-10T20:26:17.817 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:17.817 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd027fc640 10 monclient: dump: 2024-09-10T20:26:17.817 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:17.817 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 10 monclient: shutdown 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 -- 172.21.15.153:0/3482438786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 msgr2=0x7efd041366c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:17.818 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 --2- 172.21.15.153:0/3482438786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 0x7efd041366c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efd0409de10 tx=0x7efcf801c3a0 comp rx=0 tx=0).stop 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 -- 172.21.15.153:0/3482438786 shutdown_connections 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 --2- 172.21.15.153:0/3482438786 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 0x7efd041366c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 --2- 172.21.15.153:0/3482438786 >> v2:172.21.15.176:3300/0 conn(0x7efd041331e0 0x7efd041335c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 -- 172.21.15.153:0/3482438786 >> 172.21.15.153:0/3482438786 conn(0x7efd040a74a0 msgr2=0x7efd040a39a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 -- 172.21.15.153:0/3482438786 shutdown_connections 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 -- 172.21.15.153:0/3482438786 wait complete. 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 10 monclient: build_initial_monmap 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 10 monclient: monmap: 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:epoch 0 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:17.819 INFO:teuthology.misc.health.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:created 0.000000 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 librados: starting msgr at 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.798+0000 7efd09ef9640 1 librados: starting objecter 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 Processor -- start 2024-09-10T20:26:17.820 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 -- start start 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 librados: setting wanted keys 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 librados: calling monclient init 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient: init 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd041331e0 0x7efd0412e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient(hunting): picked mon.noname-b con 0x7efd041331e0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 0x7efd0412eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient(hunting): picked mon.noname-a con 0x7efd04136290 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:17.821 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd04136c00 0x7efd04138000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient(hunting): picked mon.noname-c con 0x7efd04136c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efd040a7c90 con 0x7efd04136290 2024-09-10T20:26:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efd040a7b10 con 0x7efd041331e0 2024-09-10T20:26:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efd040a0970 con 0x7efd04136c00 2024-09-10T20:26:17.822 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:17.823 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient(hunting): authenticate will time out at 937.944597s 2024-09-10T20:26:17.823 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 0x7efd0412eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:17.823 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd03fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd04136c00 0x7efd04138000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:17.823 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 0x7efd0412eda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55990/0 (socket says 172.21.15.153:55990) 2024-09-10T20:26:17.823 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 -- 172.21.15.153:0/1927557868 learned_addr learned my addr 172.21.15.153:0/1927557868 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:17.823 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient(hunting): get_auth_request con 0x7efd04136290 auth_method 0 2024-09-10T20:26:17.823 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:17.824 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:17.824 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:17.824 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd03fff640 10 monclient(hunting): get_auth_request con 0x7efd04136c00 auth_method 0 2024-09-10T20:26:17.824 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd03fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:17.824 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd03fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:17.824 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd03fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:17.824 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:17.824 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:17.824 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd03fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd03fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd03fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd037fe640 1 --2- 172.21.15.153:0/1927557868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd041331e0 0x7efd0412e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd037fe640 10 monclient(hunting): get_auth_request con 0x7efd041331e0 auth_method 0 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd037fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd037fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient(hunting): handle_auth_done global_id 4197 payload 824 2024-09-10T20:26:17.825 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 -- 172.21.15.153:0/1927557868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd04136c00 msgr2=0x7efd04138000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:17.826 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 --2- 172.21.15.153:0/1927557868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd04136c00 0x7efd04138000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:17.826 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 -- 172.21.15.153:0/1927557868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd041331e0 msgr2=0x7efd0412e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:17.826 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 --2- 172.21.15.153:0/1927557868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd041331e0 0x7efd0412e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:17.826 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:26:17.826 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 monclient: found mon.noname-a 2024-09-10T20:26:17.826 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:17.826 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:17.826 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 -- 172.21.15.153:0/1927557868 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd04139a70 con 0x7efd04136290 2024-09-10T20:26:17.826 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 -- 172.21.15.153:0/1927557868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 msgr2=0x7efd0412eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd02ffd640 1 --2- 172.21.15.153:0/1927557868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 0x7efd0412eda0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efcf801cc40 tx=0x7efcf8019140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 client.?.objecter ms_handle_connect 0x7efd04136290 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 1 -- 172.21.15.153:0/1927557868 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf80b0070 con 0x7efd04136290 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 monclient: dump: 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:17.827 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:17.828 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:17.828 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:17.828 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:17.828 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:17.828 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:17.828 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:17.828 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:17.828 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 monclient: _finish_auth 0 2024-09-10T20:26:17.829 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:17.829 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:17.829 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 1 -- 172.21.15.153:0/1927557868 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf80b1040 con 0x7efd04136290 2024-09-10T20:26:17.829 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 5 monclient: authenticate success, global_id 4197 2024-09-10T20:26:17.829 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient: _renew_subs 2024-09-10T20:26:17.829 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:17.829 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0413b530 con 0x7efd04136290 2024-09-10T20:26:17.829 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:17.830 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 1 -- 172.21.15.153:0/1927557868 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf80aa9e0 con 0x7efd04136290 2024-09-10T20:26:17.830 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:17.830 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:17.830 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efcebfff640 10 monclient: dump: 2024-09-10T20:26:17.830 INFO:teuthology.misc.health.smithi153.stderr:epoch 1 2024-09-10T20:26:17.830 INFO:teuthology.misc.health.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:17.830 INFO:teuthology.misc.health.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:17.830 INFO:teuthology.misc.health.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:17.831 INFO:teuthology.misc.health.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:17.831 INFO:teuthology.misc.health.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:17.831 INFO:teuthology.misc.health.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:17.831 INFO:teuthology.misc.health.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:17.831 INFO:teuthology.misc.health.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:17.831 INFO:teuthology.misc.health.smithi153.stderr: 2024-09-10T20:26:17.831 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd01ffb640 4 set_mon_vals no callback set 2024-09-10T20:26:17.831 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 client.4197.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient: _renew_subs 2024-09-10T20:26:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd0413e2c0 con 0x7efd04136290 2024-09-10T20:26:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.802+0000 7efd09ef9640 1 librados: init done 2024-09-10T20:26:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efd09ef9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efd09ef9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efd09ef9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd04133b90 con 0x7efd04136290 2024-09-10T20:26:17.832 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efcebfff640 1 -- 172.21.15.153:0/1927557868 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7efcf80aab80 con 0x7efd04136290 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efcebfff640 1 --2- 172.21.15.153:0/1927557868 >> v2:172.21.15.176:6812/3003473398 conn(0x7efcf003cf20 0x7efcf003f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efcebfff640 1 -- 172.21.15.153:0/1927557868 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 3153+0+0 (secure 0 0 0) 0x7efcf80e90b0 con 0x7efd04136290 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efcebfff640 10 client.4197.objecter ms_dispatch 0x7efd04000ba0 osd_map(16..16 src has 1..16) 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efcebfff640 3 client.4197.objecter handle_osd_map got epochs [16,16] > 0 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efcebfff640 3 client.4197.objecter handle_osd_map decoding full epoch 16 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efcebfff640 20 client.4197.objecter dump_active .. 0 homeless 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efd037fe640 1 --2- 172.21.15.153:0/1927557868 >> v2:172.21.15.176:6812/3003473398 conn(0x7efcf003cf20 0x7efcf003f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.806+0000 7efd037fe640 10 monclient: get_auth_request con 0x7efcf003cf20 auth_method 0 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.810+0000 7efd037fe640 1 --2- 172.21.15.153:0/1927557868 >> v2:172.21.15.176:6812/3003473398 conn(0x7efcf003cf20 0x7efcf003f3e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efcf4006fd0 tx=0x7efcf4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:17.833 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.810+0000 7efcebfff640 10 client.4197.objecter ms_handle_connect 0x7efcf003cf20 2024-09-10T20:26:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.822+0000 7efcebfff640 1 -- 172.21.15.153:0/1927557868 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7efcf80160b0 con 0x7efd04136290 2024-09-10T20:26:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.822+0000 7efcebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:17.834 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.822+0000 7efcebfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:17.998 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.990+0000 7efd09ef9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:17.998 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.990+0000 7efd09ef9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:17.998 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.990+0000 7efd09ef9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:17.998 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.990+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efd040692d0 con 0x7efd04136290 2024-09-10T20:26:17.999 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efcebfff640 1 -- 172.21.15.153:0/1927557868 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+9081 (secure 0 0 0) 0x7efcf80beb10 con 0x7efd04136290 2024-09-10T20:26:17.999 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efcebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:17.999 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efcebfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:17.999 INFO:teuthology.misc.health.smithi153.stdout: 2024-09-10T20:26:17.999 INFO:teuthology.misc.health.smithi153.stdout:{"epoch":16,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","created":"2024-09-10T20:25:59.587062+0000","modified":"2024-09-10T20:26:15.313553+0000","last_up_change":"2024-09-10T20:26:12.282988+0000","last_in_change":"2024-09-10T20:26:02.414479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:26:13.125072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"faf817a9-b230-4970-b8c5-e5637fd29193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4216589932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":4216589932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":4216589932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4216589932}]},"public_addr":"172.21.15.153:6800/4216589932","cluster_addr":"172.21.15.153:6801/4216589932","heartbeat_back_addr":"172.21.15.153:6803/4216589932","heartbeat_front_addr":"172.21.15.153:6802/4216589932","state":["exists","up"]},{"osd":1,"uuid":"f85962f8-8bd9-4e26-8d66-49b6528617a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1715391555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":1715391555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1715391555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1715391555}]},"public_addr":"172.21.15.153:6804/1715391555","cluster_addr":"172.21.15.153:6805/1715391555","heartbeat_back_addr":"172.21.15.153:6807/1715391555","heartbeat_front_addr":"172.21.15.153:6806/1715391555","state":["exists","up"]},{"osd":2,"uuid":"18d54b55-1393-42b9-9fcc-84202659cf31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1626005750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":1626005750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":1626005750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1626005750}]},"public_addr":"172.21.15.153:6808/1626005750","cluster_addr":"172.21.15.153:6809/1626005750","heartbeat_back_addr":"172.21.15.153:6811/1626005750","heartbeat_front_addr":"172.21.15.153:6810/1626005750","state":["exists","up"]},{"osd":3,"uuid":"5149b9c4-35d7-4ffb-9fac-7b3d9b660101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2953083416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":2953083416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6811","nonce":2953083416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2953083416}]},"public_addr":"172.21.15.176:6808/2953083416","cluster_addr":"172.21.15.176:6809/2953083416","heartbeat_back_addr":"172.21.15.176:6811/2953083416","heartbeat_front_addr":"172.21.15.176:6810/2953083416","state":["exists","up"]},{"osd":4,"uuid":"61e0c0cc-4301-4947-9f1d-838d2d7bbfea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4030285519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6805","nonce":4030285519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6807","nonce":4030285519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4030285519}]},"public_addr":"172.21.15.176:6804/4030285519","cluster_addr":"172.21.15.176:6805/4030285519","heartbeat_back_addr":"172.21.15.176:6807/4030285519","heartbeat_front_addr":"172.21.15.176:6806/4030285519","state":["exists","up"]},{"osd":5,"uuid":"2feacd82-c45a-4d90-8210-ff9faecc2d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":2372911183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6801","nonce":2372911183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6803","nonce":2372911183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2372911183}]},"public_addr":"172.21.15.176:6800/2372911183","cluster_addr":"172.21.15.176:6801/2372911183","heartbeat_back_addr":"172.21.15.176:6803/2372911183","heartbeat_front_addr":"172.21.15.176:6802/2372911183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.301670+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:10.037323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.733818+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364748+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.173071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:26:18.001 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 10 librados: watch_flush enter 2024-09-10T20:26:18.001 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 10 librados: watch_flush exit 2024-09-10T20:26:18.001 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 20 client.4197.objecter shutdown clearing up homeless session... 2024-09-10T20:26:18.001 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 10 client.4197.objecter successfully canceled tick 2024-09-10T20:26:18.001 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 >> v2:172.21.15.176:6812/3003473398 conn(0x7efcf003cf20 msgr2=0x7efcf003f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.001 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 --2- 172.21.15.153:0/1927557868 >> v2:172.21.15.176:6812/3003473398 conn(0x7efcf003cf20 0x7efcf003f3e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efcf4006fd0 tx=0x7efcf4008040 comp rx=0 tx=0).stop 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 10 monclient: shutdown 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 msgr2=0x7efd0412eda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 --2- 172.21.15.153:0/1927557868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 0x7efd0412eda0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efcf801cc40 tx=0x7efcf8019140 comp rx=0 tx=0).stop 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 shutdown_connections 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 --2- 172.21.15.153:0/1927557868 >> v2:172.21.15.176:6812/3003473398 conn(0x7efcf003cf20 0x7efcf003f3e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 --2- 172.21.15.153:0/1927557868 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efd04136c00 0x7efd04138000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 --2- 172.21.15.153:0/1927557868 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efd04136290 0x7efd0412eda0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 --2- 172.21.15.153:0/1927557868 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efd041331e0 0x7efd0412e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.002 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 >> 172.21.15.153:0/1927557868 conn(0x7efd040a74a0 msgr2=0x7efd040a4da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:18.003 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 shutdown_connections 2024-09-10T20:26:18.003 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 -- 172.21.15.153:0/1927557868 wait complete. 2024-09-10T20:26:18.003 INFO:teuthology.misc.health.smithi153.stderr:2024-09-10T20:26:17.994+0000 7efd09ef9640 1 librados: shutdown 2024-09-10T20:26:18.016 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-10T20:26:18.016 INFO:tasks.ceph:Creating RBD pool 2024-09-10T20:26:18.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-10T20:26:18.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.130+0000 7ff522cec640 10 monclient: get_monmap_and_config 2024-09-10T20:26:18.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.130+0000 7ff522cec640 10 monclient: build_initial_monmap 2024-09-10T20:26:18.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.130+0000 7ff522cec640 10 monclient: monmap: 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:18.137696+0000 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:18.137696+0000 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:18.139 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:18.140 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 1 Processor -- start 2024-09-10T20:26:18.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 1 -- start start 2024-09-10T20:26:18.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient: init 2024-09-10T20:26:18.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff51c0a0c00 0x7ff51c09ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient(hunting): picked mon.noname-a con 0x7ff51c0a0c00 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff51c09e710 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a1db0 0x7ff51c13c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient(hunting): picked mon.noname-b con 0x7ff51c0a1db0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff51c09ef90 con 0x7ff51c09e710 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7ff51c09f270 con 0x7ff51c0a0c00 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff51c09f550 con 0x7ff51c0a1db0 2024-09-10T20:26:18.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:18.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff522cec640 10 monclient(hunting): authenticate will time out at 938.277797s 2024-09-10T20:26:18.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff51c0a0c00 0x7ff51c09ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a1db0 0x7ff51c13c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff51c0a0c00 0x7ff51c09ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56002/0 (socket says 172.21.15.153:56002) 2024-09-10T20:26:18.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 1 -- 172.21.15.153:0/2276200025 learned_addr learned my addr 172.21.15.153:0/2276200025 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:18.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 10 monclient(hunting): get_auth_request con 0x7ff51c0a0c00 auth_method 0 2024-09-10T20:26:18.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient(hunting): get_auth_request con 0x7ff51c0a1db0 auth_method 0 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:18.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient(hunting): handle_auth_done global_id 4184 payload 274 2024-09-10T20:26:18.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 1 -- 172.21.15.153:0/2276200025 >> v2:172.21.15.153:3300/0 conn(0x7ff51c0a0c00 msgr2=0x7ff51c09ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:18.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 1 --2- 172.21.15.153:0/2276200025 >> v2:172.21.15.153:3300/0 conn(0x7ff51c0a0c00 0x7ff51c09ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 1 -- 172.21.15.153:0/2276200025 >> v1:172.21.15.176:6789/0 conn(0x7ff51c09e710 legacy=0x7ff51c0a16a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:18.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient: _finish_hunting 0 2024-09-10T20:26:18.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 1 monclient: found mon.noname-b 2024-09-10T20:26:18.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:18.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:18.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff521289640 1 -- 172.21.15.153:0/2276200025 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51c0a0fe0 con 0x7ff51c0a1db0 2024-09-10T20:26:18.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.134+0000 7ff520a88640 1 --2- 172.21.15.153:0/2276200025 >> v2:172.21.15.153:3300/0 conn(0x7ff51c0a0c00 0x7ff51c09ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:18.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff521289640 1 -- 172.21.15.153:0/2276200025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a1db0 msgr2=0x7ff51c13c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:18.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff521289640 1 --2- 172.21.15.153:0/2276200025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a1db0 0x7ff51c13c4b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff514009d90 tx=0x7ff51401c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4bde1feb188bcf59 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:18.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 1 -- 172.21.15.153:0/2276200025 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5140ac070 con 0x7ff51c0a1db0 2024-09-10T20:26:18.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 10 monclient: dump: 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:18.156 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 1 -- 172.21.15.153:0/2276200025 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5140a6840 con 0x7ff51c0a1db0 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 1 -- 172.21.15.153:0/2276200025 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5140a6c10 con 0x7ff51c0a1db0 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 5 monclient: authenticate success, global_id 4184 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 4 set_mon_vals no callback set 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:18.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff51b7fe640 10 monclient: dump: 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 10 monclient: shutdown 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:18.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 1 -- 172.21.15.153:0/2276200025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a1db0 msgr2=0x7ff51c13c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 1 --2- 172.21.15.153:0/2276200025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a1db0 0x7ff51c13c4b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff514009d90 tx=0x7ff51401c830 comp rx=0 tx=0).stop 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 1 -- 172.21.15.153:0/2276200025 shutdown_connections 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 1 --2- 172.21.15.153:0/2276200025 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a1db0 0x7ff51c13c4b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 1 --2- 172.21.15.153:0/2276200025 >> v2:172.21.15.153:3300/0 conn(0x7ff51c0a0c00 0x7ff51c09ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 1 -- 172.21.15.153:0/2276200025 >> 172.21.15.153:0/2276200025 conn(0x7ff51c0a7890 msgr2=0x7ff51c0a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 1 -- 172.21.15.153:0/2276200025 shutdown_connections 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.138+0000 7ff522cec640 1 -- 172.21.15.153:0/2276200025 wait complete. 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient: build_initial_monmap 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient: monmap: 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:18.159 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 librados: starting msgr at 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 librados: starting objecter 2024-09-10T20:26:18.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 Processor -- start 2024-09-10T20:26:18.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 -- start start 2024-09-10T20:26:18.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 librados: setting wanted keys 2024-09-10T20:26:18.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 librados: calling monclient init 2024-09-10T20:26:18.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient: init 2024-09-10T20:26:18.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:18.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:26:18.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff51c09e710 0x7ff51c126fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient(hunting): picked mon.noname-b con 0x7ff51c09e710 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:18.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a0c00 0x7ff51c1274f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient(hunting): picked mon.noname-c con 0x7ff51c0a0c00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:18.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c0a1db0 0x7ff51c127a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient(hunting): picked mon.noname-a con 0x7ff51c0a1db0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:18.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff51c13ed90 con 0x7ff51c0a1db0 2024-09-10T20:26:18.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff51c13ec10 con 0x7ff51c09e710 2024-09-10T20:26:18.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff51c13ef10 con 0x7ff51c0a0c00 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff522cec640 10 monclient(hunting): authenticate will time out at 938.285162s 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a0c00 0x7ff51c1274f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff521289640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c0a1db0 0x7ff51c127a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a0c00 0x7ff51c1274f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47320/0 (socket says 172.21.15.153:47320) 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 1 -- 172.21.15.153:0/3633527688 learned_addr learned my addr 172.21.15.153:0/3633527688 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient(hunting): get_auth_request con 0x7ff51c0a0c00 auth_method 0 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:18.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff521289640 10 monclient(hunting): get_auth_request con 0x7ff51c0a1db0 auth_method 0 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff521289640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff521289640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff521289640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff521289640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff521289640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff521289640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:18.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff520a88640 1 --2- 172.21.15.153:0/3633527688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff51c09e710 0x7ff51c126fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff520a88640 10 monclient(hunting): get_auth_request con 0x7ff51c09e710 auth_method 0 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff520a88640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff520a88640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff520a88640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient(hunting): handle_auth_done global_id 4187 payload 824 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 1 -- 172.21.15.153:0/3633527688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff51c09e710 msgr2=0x7ff51c126fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 1 --2- 172.21.15.153:0/3633527688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff51c09e710 0x7ff51c126fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 1 -- 172.21.15.153:0/3633527688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c0a1db0 msgr2=0x7ff51c127a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 1 --2- 172.21.15.153:0/3633527688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c0a1db0 0x7ff51c127a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 1 monclient: found mon.noname-c 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:18.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff51bfff640 1 -- 172.21.15.153:0/3633527688 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff51c12a6f0 con 0x7ff51c0a0c00 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.142+0000 7ff521289640 1 --2- 172.21.15.153:0/3633527688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c0a1db0 0x7ff51c127a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff51bfff640 1 -- 172.21.15.153:0/3633527688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a0c00 msgr2=0x7ff51c1274f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff51bfff640 1 --2- 172.21.15.153:0/3633527688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a0c00 0x7ff51c1274f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff51000c450 tx=0x7ff51000c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 client.?.objecter ms_handle_connect 0x7ff51c0a0c00 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 1 -- 172.21.15.153:0/3633527688 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5100119a0 con 0x7ff51c0a0c00 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 monclient: dump: 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.167 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 monclient: _finish_auth 0 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 monclient: _check_auth_tickets 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 5 monclient: authenticate success, global_id 4187 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 1 -- 172.21.15.153:0/3633527688 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff510017070 con 0x7ff51c0a0c00 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 10 monclient: _renew_subs 2024-09-10T20:26:18.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 1 -- 172.21.15.153:0/3633527688 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff510011da0 con 0x7ff51c0a0c00 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff51c12a9d0 con 0x7ff51c0a0c00 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff51affd640 4 set_mon_vals no callback set 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 monclient: dump: 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:18.169 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 10 client.4187.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 10 monclient: _renew_subs 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff51c12acb0 con 0x7ff51c0a0c00 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 1 librados: init done 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 1 -- 172.21.15.153:0/3633527688 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7ff51001c460 con 0x7ff51c0a0c00 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff51c09ffb0 con 0x7ff51c0a0c00 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 1 --2- 172.21.15.153:0/3633527688 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff50803cef0 0x7ff50803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 1 -- 172.21.15.153:0/3633527688 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(16..16 src has 1..16) ==== 3153+0+0 (secure 0 0 0) 0x7ff51004ee60 con 0x7ff51c0a0c00 2024-09-10T20:26:18.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 10 client.4187.objecter ms_dispatch 0x7ff51c000ba0 osd_map(16..16 src has 1..16) 2024-09-10T20:26:18.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 3 client.4187.objecter handle_osd_map got epochs [16,16] > 0 2024-09-10T20:26:18.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 3 client.4187.objecter handle_osd_map decoding full epoch 16 2024-09-10T20:26:18.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff5197fa640 20 client.4187.objecter dump_active .. 0 homeless 2024-09-10T20:26:18.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff520a88640 1 --2- 172.21.15.153:0/3633527688 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff50803cef0 0x7ff50803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.146+0000 7ff520a88640 10 monclient: get_auth_request con 0x7ff50803cef0 auth_method 0 2024-09-10T20:26:18.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.150+0000 7ff5197fa640 10 client.4187.objecter ms_handle_connect 0x7ff50803cef0 2024-09-10T20:26:18.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.150+0000 7ff520a88640 1 --2- 172.21.15.153:0/3633527688 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff50803cef0 0x7ff50803f3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00a7d0 tx=0x7ff50c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:18.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.150+0000 7ff5197fa640 1 -- 172.21.15.153:0/3633527688 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7ff51001c930 con 0x7ff51c0a0c00 2024-09-10T20:26:18.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.150+0000 7ff5197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:18.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.150+0000 7ff5197fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:18.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.310+0000 7ff522cec640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-10T20:26:18.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.310+0000 7ff522cec640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-10T20:26:18.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.310+0000 7ff522cec640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:18.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.310+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7ff51c06fb30 con 0x7ff51c0a0c00 2024-09-10T20:26:18.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff5197fa640 1 -- 172.21.15.153:0/3633527688 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v17) ==== 109+0+0 (secure 0 0 0) 0x7ff51004deb0 con 0x7ff51c0a0c00 2024-09-10T20:26:18.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff5197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-10T20:26:18.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff5197fa640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-10T20:26:18.348 INFO:teuthology.orchestra.run.smithi153.stderr:pool 'rbd' created 2024-09-10T20:26:18.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 10 librados: watch_flush enter 2024-09-10T20:26:18.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 10 librados: watch_flush exit 2024-09-10T20:26:18.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 20 client.4187.objecter shutdown clearing up homeless session... 2024-09-10T20:26:18.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 10 client.4187.objecter successfully canceled tick 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff50803cef0 msgr2=0x7ff50803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 --2- 172.21.15.153:0/3633527688 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff50803cef0 0x7ff50803f3b0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ff50c00a7d0 tx=0x7ff50c008040 comp rx=0 tx=0).stop 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 10 monclient: shutdown 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a0c00 msgr2=0x7ff51c1274f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 --2- 172.21.15.153:0/3633527688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a0c00 0x7ff51c1274f0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff51000c450 tx=0x7ff51000c910 comp rx=0 tx=0).stop 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 shutdown_connections 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 --2- 172.21.15.153:0/3633527688 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff51c0a1db0 0x7ff51c127a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 --2- 172.21.15.153:0/3633527688 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff50803cef0 0x7ff50803f3b0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 --2- 172.21.15.153:0/3633527688 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff51c0a0c00 0x7ff51c1274f0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 --2- 172.21.15.153:0/3633527688 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff51c09e710 0x7ff51c126fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 >> 172.21.15.153:0/3633527688 conn(0x7ff51c0a7890 msgr2=0x7ff51c072300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 shutdown_connections 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.342+0000 7ff522cec640 1 -- 172.21.15.153:0/3633527688 wait complete. 2024-09-10T20:26:18.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.346+0000 7ff522cec640 1 librados: shutdown 2024-09-10T20:26:18.366 DEBUG:teuthology.orchestra.run.smithi153:> rbd --cluster ceph pool init rbd 2024-09-10T20:26:18.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.422+0000 7f7b031ea100 10 monclient: get_monmap_and_config 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.422+0000 7f7b031ea100 10 monclient: build_initial_monmap 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.422+0000 7f7b031ea100 10 monclient: monmap: 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:18.429097+0000 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:18.429097+0000 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:18.430 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:18.431 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 1 Processor -- start 2024-09-10T20:26:18.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 1 -- start start 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient: init 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient: _reopen_session rank -1 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient: _add_conns ranks=[2,3,4,0,1] 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c2fbde4560 0x55c2fbde4940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient(hunting): picked mon.noname-b con 0x55c2fbde4560 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 1 --2- >> v2:172.21.15.153:3300/0 conn(0x55c2fbddd600 0x55c2fbde4e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient(hunting): picked mon.noname-a con 0x55c2fbddd600 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 1 --2- >> v2:172.21.15.176:3300/0 conn(0x55c2fbe78330 0x55c2fbe78760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient(hunting): picked mon.noname-c con 0x55c2fbe78330 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:18.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x55c2fbcaf160 con 0x55c2fbddd600 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x55c2fbcbfd30 con 0x55c2fbe78330 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55c2fbcb2f70 con 0x55c2fbde4560 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient(hunting): _renew_subs 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b031ea100 10 monclient(hunting): authenticate will time out at 938.568965s 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x55c2fbddd600 0x55c2fbde4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x55c2fbddd600 0x55c2fbde4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56026/0 (socket says 172.21.15.153:56026) 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 1 -- 172.21.15.153:0/4183818275 learned_addr learned my addr 172.21.15.153:0/4183818275 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:18.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b01cf3640 1 --2- 172.21.15.153:0/4183818275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c2fbde4560 0x55c2fbde4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient(hunting): get_auth_request con 0x55c2fbddd600 auth_method 0 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b01cf3640 10 monclient(hunting): get_auth_request con 0x55c2fbde4560 auth_method 0 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b01cf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b01cf3640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b01cf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b01cf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b01cf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:18.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b01cf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b02f55640 1 --2- 172.21.15.153:0/4183818275 >> v2:172.21.15.176:3300/0 conn(0x55c2fbe78330 0x55c2fbe78760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient(hunting): handle_auth_done global_id 4206 payload 274 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 1 -- 172.21.15.153:0/4183818275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c2fbde4560 msgr2=0x55c2fbde4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 1 --2- 172.21.15.153:0/4183818275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c2fbde4560 0x55c2fbde4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 1 -- 172.21.15.153:0/4183818275 >> v2:172.21.15.176:3300/0 conn(0x55c2fbe78330 msgr2=0x55c2fbe78760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 1 --2- 172.21.15.153:0/4183818275 >> v2:172.21.15.176:3300/0 conn(0x55c2fbe78330 0x55c2fbe78760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient: _finish_hunting 0 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 1 monclient: found mon.noname-a 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:18.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b014f2640 1 -- 172.21.15.153:0/4183818275 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2fbc2dc80 con 0x55c2fbddd600 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b02f55640 10 monclient: get_auth_request con 0x55c2fbe78330 auth_method 0 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.426+0000 7f7b02f55640 1 --2- 172.21.15.153:0/4183818275 >> v2:172.21.15.176:3300/0 conn(0x55c2fbe78330 0x55c2fbe78760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b014f2640 1 -- 172.21.15.153:0/4183818275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 msgr2=0x55c2fbde4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b014f2640 1 --2- 172.21.15.153:0/4183818275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 0x55c2fbde4e80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x55c2fbcb6b00 tx=0x7f7af801b3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=189de67f6a983a42 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 1 -- 172.21.15.153:0/4183818275 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80b1070 con 0x55c2fbddd600 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 10 monclient: dump: 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.446 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.447 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:18.447 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.447 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:18.447 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:18.447 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:18.447 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 10 monclient: _finish_auth 0 2024-09-10T20:26:18.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 10 monclient: _check_auth_tickets 2024-09-10T20:26:18.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:18.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 1 -- 172.21.15.153:0/4183818275 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af80a8070 con 0x55c2fbddd600 2024-09-10T20:26:18.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:18.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 1 -- 172.21.15.153:0/4183818275 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af80ac880 con 0x55c2fbddd600 2024-09-10T20:26:18.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 5 monclient: authenticate success, global_id 4206 2024-09-10T20:26:18.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 10 monclient: get_monmap_and_config success 2024-09-10T20:26:18.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 4 set_mon_vals no callback set 2024-09-10T20:26:18.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:18.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:18.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b00cf1640 10 monclient: dump: 2024-09-10T20:26:18.449 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:18.450 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.450 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.450 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.450 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:18.450 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.450 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:18.450 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:18.451 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:18.451 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 10 monclient: shutdown 2024-09-10T20:26:18.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:18.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 1 -- 172.21.15.153:0/4183818275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 msgr2=0x55c2fbde4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 1 --2- 172.21.15.153:0/4183818275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 0x55c2fbde4e80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x55c2fbcb6b00 tx=0x7f7af801b3a0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 1 -- 172.21.15.153:0/4183818275 shutdown_connections 2024-09-10T20:26:18.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 1 --2- 172.21.15.153:0/4183818275 >> v2:172.21.15.176:3300/0 conn(0x55c2fbe78330 0x55c2fbe78760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 1 --2- 172.21.15.153:0/4183818275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 0x55c2fbde4e80 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 1 --2- 172.21.15.153:0/4183818275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c2fbde4560 0x55c2fbde4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 1 -- 172.21.15.153:0/4183818275 >> 172.21.15.153:0/4183818275 conn(0x55c2fbde5650 msgr2=0x55c2fbde5a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:18.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 1 -- 172.21.15.153:0/4183818275 shutdown_connections 2024-09-10T20:26:18.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.430+0000 7f7b031ea100 1 -- 172.21.15.153:0/4183818275 wait complete. 2024-09-10T20:26:18.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient: build_initial_monmap 2024-09-10T20:26:18.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient: monmap: 2024-09-10T20:26:18.453 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:18.453 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:18.453 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:18.454 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:18.454 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:18.454 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.454 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:18.454 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:18.454 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:18.455 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 librados: starting msgr at 2024-09-10T20:26:18.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 librados: starting objecter 2024-09-10T20:26:18.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 Processor -- start 2024-09-10T20:26:18.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 -- start start 2024-09-10T20:26:18.455 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 librados: setting wanted keys 2024-09-10T20:26:18.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 librados: calling monclient init 2024-09-10T20:26:18.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient: init 2024-09-10T20:26:18.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient: _reopen_session rank -1 2024-09-10T20:26:18.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:26:18.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 0x55c2fbe8a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.456 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient(hunting): picked mon.noname-a con 0x55c2fbddd600 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c2fbde4560 0x55c2fbe8a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient(hunting): picked mon.noname-b con 0x55c2fbde4560 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c2fbe78330 0x55c2fbe8ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient(hunting): picked mon.noname-c con 0x55c2fbe78330 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.457 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x55c2fbde6470 con 0x55c2fbddd600 2024-09-10T20:26:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x55c2fbc54a70 con 0x55c2fbde4560 2024-09-10T20:26:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x55c2fbdc6500 con 0x55c2fbe78330 2024-09-10T20:26:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient(hunting): _renew_subs 2024-09-10T20:26:18.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b031ea100 10 monclient(hunting): authenticate will time out at 938.577502s 2024-09-10T20:26:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 0x55c2fbe8a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b02f55640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c2fbe78330 0x55c2fbe8ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 0x55c2fbe8a320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56028/0 (socket says 172.21.15.153:56028) 2024-09-10T20:26:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 1 -- 172.21.15.153:0/3429870644 learned_addr learned my addr 172.21.15.153:0/3429870644 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient(hunting): get_auth_request con 0x55c2fbddd600 auth_method 0 2024-09-10T20:26:18.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b02f55640 10 monclient(hunting): get_auth_request con 0x55c2fbe78330 auth_method 0 2024-09-10T20:26:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b02f55640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b02f55640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:18.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b02f55640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:18.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:18.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:18.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:18.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b02f55640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:18.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b02f55640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:18.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b02f55640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient(hunting): handle_auth_done global_id 4209 payload 824 2024-09-10T20:26:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 1 -- 172.21.15.153:0/3429870644 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c2fbe78330 msgr2=0x55c2fbe8ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 1 --2- 172.21.15.153:0/3429870644 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x55c2fbe78330 0x55c2fbe8ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 1 -- 172.21.15.153:0/3429870644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c2fbde4560 msgr2=0x55c2fbe8a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b014f2640 1 --2- 172.21.15.153:0/3429870644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c2fbde4560 0x55c2fbe8a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 1 --2- 172.21.15.153:0/3429870644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x55c2fbde4560 0x55c2fbe8a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient: _finish_hunting 0 2024-09-10T20:26:18.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 1 monclient: found mon.noname-a 2024-09-10T20:26:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:18.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.434+0000 7f7b01cf3640 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2fbe62200 con 0x55c2fbddd600 2024-09-10T20:26:18.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b01cf3640 1 -- 172.21.15.153:0/3429870644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 msgr2=0x55c2fbe8a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:18.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b01cf3640 1 --2- 172.21.15.153:0/3429870644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 0x55c2fbe8a320 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7af00087d0 tx=0x7f7af0008c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:18.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 client.?.objecter ms_handle_connect 0x55c2fbddd600 2024-09-10T20:26:18.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:18.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0017020 con 0x55c2fbddd600 2024-09-10T20:26:18.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:18.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:18.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 monclient: dump: 2024-09-10T20:26:18.467 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:18.467 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.467 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.467 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.468 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:18.468 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.468 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:18.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:18.468 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:18.468 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:18.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7af000dcc0 con 0x55c2fbddd600 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7af0013600 con 0x55c2fbddd600 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 monclient: dump: 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:18.469 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 5 monclient: authenticate success, global_id 4209 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aeffff640 4 set_mon_vals no callback set 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 monclient: _renew_subs 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x55c2fbe7f430 con 0x55c2fbddd600 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 monclient: _renew_subs 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c2fbe619c0 con 0x55c2fbddd600 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 1 librados: init done 2024-09-10T20:26:18.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 librados: wait_for_osdmap waiting 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f7af001a020 con 0x55c2fbddd600 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ae003b020 0x7f7ae003d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 3495+0+0 (secure 0 0 0) 0x7f7af004e8a0 con 0x55c2fbddd600 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 osd_map(17..17 src has 1..17) 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 3 client.4209.objecter handle_osd_map got epochs [17,17] > 0 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 3 client.4209.objecter handle_osd_map decoding full epoch 17 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7aedffb640 20 client.4209.objecter dump_active .. 0 homeless 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 librados: wait_for_osdmap done waiting 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55c2fbe60ed0 con 0x55c2fbddd600 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b014f2640 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ae003b020 0x7f7ae003d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.438+0000 7f7b014f2640 10 monclient: get_auth_request con 0x7f7ae003b020 auth_method 0 2024-09-10T20:26:18.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.442+0000 7f7b014f2640 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ae003b020 0x7f7ae003d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7af80ab000 tx=0x7f7af80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:18.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:18.442+0000 7f7aedffb640 10 client.4209.objecter ms_handle_connect 0x7f7ae003b020 2024-09-10T20:26:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.342+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v18) ==== 141+0+0 (secure 0 0 0) 0x55c2fbe619c0 con 0x55c2fbddd600 2024-09-10T20:26:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.342+0000 7f7aedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-10T20:26:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.342+0000 7f7aedffb640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-10T20:26:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.342+0000 7f7b031ea100 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:19.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.342+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7af80a8070 con 0x55c2fbddd600 2024-09-10T20:26:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_get_version_reply(handle=1 version=18) ==== 24+0+0 (secure 0 0 0) 0x7f7af001c790 con 0x55c2fbddd600 2024-09-10T20:26:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aedffb640 10 monclient: handle_get_version_reply finishing 1 version 18 2024-09-10T20:26:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aeffff640 10 client.4209.objecter _get_latest_version latest 18, waiting 2024-09-10T20:26:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aeffff640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aeffff640 10 monclient: _renew_subs 2024-09-10T20:26:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aeffff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:19.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aeffff640 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=18}) -- 0x7f7ae4003ca0 con 0x55c2fbddd600 2024-09-10T20:26:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 8 ==== osd_map(18..18 src has 1..18) ==== 674+0+0 (secure 0 0 0) 0x7f7af00137e0 con 0x55c2fbddd600 2024-09-10T20:26:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aedffb640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 osd_map(18..18 src has 1..18) 2024-09-10T20:26:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aedffb640 3 client.4209.objecter handle_osd_map got epochs [18,18] > 17 2024-09-10T20:26:19.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aedffb640 3 client.4209.objecter handle_osd_map decoding incremental epoch 18 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7aedffb640 20 client.4209.objecter dump_active .. 0 homeless 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 10 librados: create oid=rbd_trash nspace= 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 10 client.4209.objecter _op_submit op 0x55c2fbe77410 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 20 client.4209.objecter _calc_target epoch 18 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 20 client.4209.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 10 client.4209.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [1,2] primary 1 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6804/1715391555 conn(0x55c2fbe91910 0x55c2fbe93d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 20 client.4209.objecter _get_session s=0x55c2fbe778c0 osd=1 3 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 10 client.4209.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.1 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 20 client.4209.objecter get_session s=0x55c2fbe778c0 osd=1 3 2024-09-10T20:26:19.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 15 client.4209.objecter _session_op_assign 1 1 2024-09-10T20:26:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 15 client.4209.objecter _send_op 1 to 2.3 on osd.1 2024-09-10T20:26:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x55c2fbe94240 con 0x55c2fbe91910 2024-09-10T20:26:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 20 client.4209.objecter put_session s=0x55c2fbe778c0 osd=1 4 2024-09-10T20:26:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b031ea100 5 client.4209.objecter 1 in flight 2024-09-10T20:26:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b02f55640 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6804/1715391555 conn(0x55c2fbe91910 0x55c2fbe93d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:19.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.350+0000 7f7b02f55640 10 monclient: get_auth_request con 0x55c2fbe91910 auth_method 0 2024-09-10T20:26:19.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.358+0000 7f7aedffb640 10 client.4209.objecter ms_handle_connect 0x55c2fbe91910 2024-09-10T20:26:19.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.358+0000 7f7b02f55640 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6804/1715391555 conn(0x55c2fbe91910 0x55c2fbe93d00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 1 -- 172.21.15.153:0/3429870644 <== osd.1 v2:172.21.15.153:6804/1715391555 1 ==== osd_op_reply(1 rbd_trash [create] v18'1 uv1 ondisk = 0) ==== 153+0+0 (crc 0 0 0) 0x7f7af400b990 con 0x55c2fbe91910 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 osd_op_reply(1 rbd_trash [create] v18'1 uv1 ondisk = 0) 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 10 client.4209.objecter in handle_osd_op_reply 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 7 client.4209.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 10 client.4209.objecter op 0 rval 0 len 0 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 15 client.4209.objecter handle_osd_op_reply completed tid 1 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 15 client.4209.objecter _finish_op 1 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 20 client.4209.objecter put_session s=0x55c2fbe778c0 osd=1 4 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 15 client.4209.objecter _session_op_remove 1 1 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b02f55640 5 client.4209.objecter 0 in flight 2024-09-10T20:26:19.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.362+0000 7f7b031ea100 10 librados: Objecter returned from create r=0 2024-09-10T20:26:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 10 client.4209.objecter _op_submit op 0x55c2fbee6f70 2024-09-10T20:26:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 20 client.4209.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:26:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 20 client.4209.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-10T20:26:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 10 client.4209.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [0,4] primary 0 2024-09-10T20:26:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6800/4216589932 conn(0x55c2fbee7b20 0x55c2fbee9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 20 client.4209.objecter _get_session s=0x55c2fbee74a0 osd=0 3 2024-09-10T20:26:19.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 10 client.4209.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.0 2024-09-10T20:26:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 20 client.4209.objecter get_session s=0x55c2fbee74a0 osd=0 3 2024-09-10T20:26:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 15 client.4209.objecter _session_op_assign 0 2 2024-09-10T20:26:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 15 client.4209.objecter _send_op 2 to 2.2 on osd.0 2024-09-10T20:26:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 --> v2:172.21.15.153:6800/4216589932 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) -- 0x55c2fbeea560 con 0x55c2fbee7b20 2024-09-10T20:26:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 20 client.4209.objecter put_session s=0x55c2fbee74a0 osd=0 4 2024-09-10T20:26:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b031ea100 5 client.4209.objecter 1 in flight 2024-09-10T20:26:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b01cf3640 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6800/4216589932 conn(0x55c2fbee7b20 0x55c2fbee9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:19.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.366+0000 7f7b01cf3640 10 monclient: get_auth_request con 0x55c2fbee7b20 auth_method 0 2024-09-10T20:26:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.370+0000 7f7b01cf3640 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6800/4216589932 conn(0x55c2fbee7b20 0x55c2fbee9fb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:19.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.370+0000 7f7aedffb640 10 client.4209.objecter ms_handle_connect 0x55c2fbee7b20 2024-09-10T20:26:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 1 -- 172.21.15.153:0/3429870644 <== osd.0 v2:172.21.15.153:6800/4216589932 1 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f7af0013ac0 con 0x55c2fbee7b20 2024-09-10T20:26:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-10T20:26:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 10 client.4209.objecter in handle_osd_op_reply 2024-09-10T20:26:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 7 client.4209.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-10T20:26:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 10 client.4209.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-10T20:26:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 10 client.4209.objecter op 0 rval 0 len 0 2024-09-10T20:26:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 15 client.4209.objecter handle_osd_op_reply completed tid 2 2024-09-10T20:26:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 15 client.4209.objecter _finish_op 2 2024-09-10T20:26:19.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 20 client.4209.objecter put_session s=0x55c2fbee74a0 osd=0 4 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 15 client.4209.objecter _session_op_remove 0 2 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 5 client.4209.objecter 0 in flight 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 10 client.4209.objecter _op_submit op 0x55c2fbee6100 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 20 client.4209.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 20 client.4209.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 10 client.4209.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [0,4] primary 0 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 20 client.4209.objecter _get_session s=0x55c2fbee74a0 osd=0 3 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 10 client.4209.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.0 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 20 client.4209.objecter get_session s=0x55c2fbee74a0 osd=0 3 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 15 client.4209.objecter _session_op_assign 0 3 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 15 client.4209.objecter _send_op 3 to 2.2 on osd.0 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 --> v2:172.21.15.153:6800/4216589932 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e18) -- 0x55c2fbee65b0 con 0x55c2fbee7b20 2024-09-10T20:26:19.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 20 client.4209.objecter put_session s=0x55c2fbee74a0 osd=0 4 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b031ea100 5 client.4209.objecter 1 in flight 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 1 -- 172.21.15.153:0/3429870644 <== osd.0 v2:172.21.15.153:6800/4216589932 2 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (crc 0 0 0) 0x7f7af0013ac0 con 0x55c2fbee7b20 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 10 client.4209.objecter in handle_osd_op_reply 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 7 client.4209.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 10 client.4209.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 10 client.4209.objecter op 0 rval 0 len 0 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 15 client.4209.objecter handle_osd_op_reply completed tid 3 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 15 client.4209.objecter _finish_op 3 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 20 client.4209.objecter put_session s=0x55c2fbee74a0 osd=0 4 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 15 client.4209.objecter _session_op_remove 0 3 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7b01cf3640 5 client.4209.objecter 0 in flight 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7aef7fe640 10 client.4209.objecter allocate_selfmanaged_snap; pool: 2 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7aef7fe640 10 client.4209.objecter pool_op_submit 4 2024-09-10T20:26:19.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7aef7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:19.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:19.374+0000 7f7aef7fe640 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f7ad80030e0 con 0x55c2fbddd600 2024-09-10T20:26:20.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 9 ==== pool_op_reply(tid 4 (0) Success v19) ==== 55+0+0 (secure 0 0 0) 0x7f7af0021030 con 0x55c2fbddd600 2024-09-10T20:26:20.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 pool_op_reply(tid 4 (0) Success v19) 2024-09-10T20:26:20.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 10 client.4209.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v19) 2024-09-10T20:26:20.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 10 client.4209.objecter have request 4 at 0x7f7ad8003070 Op: create unmanaged snap 2024-09-10T20:26:20.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 20 client.4209.objecter waiting for client to reach epoch 19 before calling back 2024-09-10T20:26:20.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:20.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 10 monclient: _renew_subs 2024-09-10T20:26:20.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:20.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=19}) -- 0x7f7ae0040f40 con 0x55c2fbddd600 2024-09-10T20:26:20.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 10 client.4209.objecter done 2024-09-10T20:26:20.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 10 ==== osd_map(19..19 src has 1..19) ==== 626+0+0 (secure 0 0 0) 0x7f7af0006cb0 con 0x55c2fbddd600 2024-09-10T20:26:20.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 osd_map(19..19 src has 1..19) 2024-09-10T20:26:20.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 3 client.4209.objecter handle_osd_map got epochs [19,19] > 18 2024-09-10T20:26:20.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 3 client.4209.objecter handle_osd_map decoding incremental epoch 19 2024-09-10T20:26:20.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aedffb640 20 client.4209.objecter dump_active .. 0 homeless 2024-09-10T20:26:20.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 20 librados: queue_aio_write 0x55c2fbee2500 completion 0x55c2fbc34fb0 write_seq 1 2024-09-10T20:26:20.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 10 client.4209.objecter _op_submit op 0x7f7ae4006390 2024-09-10T20:26:20.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 20 client.4209.objecter _calc_target epoch 19 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 20 client.4209.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 10 client.4209.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [0,4] primary 0 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 20 client.4209.objecter _get_session s=0x55c2fbee74a0 osd=0 3 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 10 client.4209.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.0 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 20 client.4209.objecter get_session s=0x55c2fbee74a0 osd=0 3 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 15 client.4209.objecter _session_op_assign 0 5 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 15 client.4209.objecter _send_op 5 to 2.2 on osd.0 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 1 -- 172.21.15.153:0/3429870644 --> v2:172.21.15.153:6800/4216589932 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f7ae40069d0 con 0x55c2fbee7b20 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 20 client.4209.objecter put_session s=0x55c2fbee74a0 osd=0 4 2024-09-10T20:26:20.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.358+0000 7f7aeffff640 5 client.4209.objecter 1 in flight 2024-09-10T20:26:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 1 -- 172.21.15.153:0/3429870644 <== osd.0 v2:172.21.15.153:6800/4216589932 3 ==== osd_op_reply(5 rbd_info [create,write 0~8] v19'1 uv1 ondisk = 0) ==== 194+0+0 (crc 0 0 0) 0x7f7af0013ac0 con 0x55c2fbee7b20 2024-09-10T20:26:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 osd_op_reply(5 rbd_info [create,write 0~8] v19'1 uv1 ondisk = 0) 2024-09-10T20:26:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 10 client.4209.objecter in handle_osd_op_reply 2024-09-10T20:26:20.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 7 client.4209.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 10 client.4209.objecter op 0 rval 0 len 0 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 10 client.4209.objecter op 1 rval 0 len 0 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 15 client.4209.objecter handle_osd_op_reply completed tid 5 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 15 client.4209.objecter _finish_op 5 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 20 client.4209.objecter put_session s=0x55c2fbee74a0 osd=0 4 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 15 client.4209.objecter _session_op_remove 0 5 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 5 client.4209.objecter 0 in flight 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7b01cf3640 20 librados: complete_aio_write 0x55c2fbc34fb0 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7aef7fe640 10 client.4209.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7aef7fe640 10 client.4209.objecter pool_op_submit 6 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7aef7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:20.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:20.366+0000 7f7aef7fe640 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- pool_op(delete unmanaged snap pool 2 tid 6 name v19) -- 0x7f7ad8004530 con 0x55c2fbddd600 2024-09-10T20:26:21.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 11 ==== pool_op_reply(tid 6 (0) Success v20) ==== 43+0+0 (secure 0 0 0) 0x7f7af004c020 con 0x55c2fbddd600 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 pool_op_reply(tid 6 (0) Success v20) 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 10 client.4209.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v20) 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 10 client.4209.objecter have request 6 at 0x7f7ad8003070 Op: delete unmanaged snap 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 20 client.4209.objecter waiting for client to reach epoch 20 before calling back 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 10 client.4209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 10 monclient: _renew_subs 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=20}) -- 0x7f7ae0041110 con 0x55c2fbddd600 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 10 client.4209.objecter done 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 1 -- 172.21.15.153:0/3429870644 <== mon.0 v2:172.21.15.153:3300/0 12 ==== osd_map(20..20 src has 1..20) ==== 654+0+0 (secure 0 0 0) 0x7f7af004c200 con 0x55c2fbddd600 2024-09-10T20:26:21.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 osd_map(20..20 src has 1..20) 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 3 client.4209.objecter handle_osd_map got epochs [20,20] > 19 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 3 client.4209.objecter handle_osd_map decoding incremental epoch 20 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aedffb640 20 client.4209.objecter dump_active .. 0 homeless 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 20 librados: queue_aio_write 0x55c2fbee2500 completion 0x7f7ad8001d70 write_seq 2 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 10 client.4209.objecter _op_submit op 0x7f7ae4006390 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 20 client.4209.objecter _calc_target epoch 20 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 20 client.4209.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 10 client.4209.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [0,4] primary 0 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 20 client.4209.objecter _get_session s=0x55c2fbee74a0 osd=0 3 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 10 client.4209.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.0 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 20 client.4209.objecter get_session s=0x55c2fbee74a0 osd=0 3 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 15 client.4209.objecter _session_op_assign 0 7 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 15 client.4209.objecter _send_op 7 to 2.2 on osd.0 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 1 -- 172.21.15.153:0/3429870644 --> v2:172.21.15.153:6800/4216589932 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20) -- 0x7f7ae4006d70 con 0x55c2fbee7b20 2024-09-10T20:26:21.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 20 client.4209.objecter put_session s=0x55c2fbee74a0 osd=0 4 2024-09-10T20:26:21.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.374+0000 7f7aeffff640 5 client.4209.objecter 1 in flight 2024-09-10T20:26:21.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 1 -- 172.21.15.153:0/3429870644 <== osd.0 v2:172.21.15.153:6800/4216589932 4 ==== osd_op_reply(7 rbd_info [write 0~19] v20'2 uv2 ondisk = 0) ==== 152+0+0 (crc 0 0 0) 0x7f7af0013ac0 con 0x55c2fbee7b20 2024-09-10T20:26:21.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 10 client.4209.objecter ms_dispatch 0x55c2fbcc7a30 osd_op_reply(7 rbd_info [write 0~19] v20'2 uv2 ondisk = 0) 2024-09-10T20:26:21.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 10 client.4209.objecter in handle_osd_op_reply 2024-09-10T20:26:21.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 7 client.4209.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 10 client.4209.objecter op 0 rval 0 len 0 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 15 client.4209.objecter handle_osd_op_reply completed tid 7 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 15 client.4209.objecter _finish_op 7 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 20 client.4209.objecter put_session s=0x55c2fbee74a0 osd=0 4 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 15 client.4209.objecter _session_op_remove 0 7 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 5 client.4209.objecter 0 in flight 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b01cf3640 20 librados: complete_aio_write 0x7f7ad8001d70 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 10 librados: watch_flush enter 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 10 librados: watch_flush exit 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 10 client.4209.objecter close_session for osd.0 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6800/4216589932 conn(0x55c2fbee7b20 msgr2=0x55c2fbee9fb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6800/4216589932 conn(0x55c2fbee7b20 0x55c2fbee9fb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 20 client.4209.objecter put_session s=0x55c2fbee74a0 osd=0 1 2024-09-10T20:26:21.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 10 client.4209.objecter close_session for osd.1 2024-09-10T20:26:21.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6804/1715391555 conn(0x55c2fbe91910 msgr2=0x55c2fbe93d00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.153:6804/1715391555 conn(0x55c2fbe91910 0x55c2fbe93d00 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 20 client.4209.objecter put_session s=0x55c2fbe778c0 osd=1 1 2024-09-10T20:26:21.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 20 client.4209.objecter shutdown clearing up homeless session... 2024-09-10T20:26:21.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 10 client.4209.objecter successfully canceled tick 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ae003b020 msgr2=0x7f7ae003d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 --2- 172.21.15.153:0/3429870644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ae003b020 0x7f7ae003d4e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7af80ab000 tx=0x7f7af80a6000 comp rx=0 tx=0).stop 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 10 monclient: shutdown 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 msgr2=0x55c2fbe8a320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b02f55640 1 -- 172.21.15.153:0/3429870644 reap_dead start 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 --2- 172.21.15.153:0/3429870644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 0x55c2fbe8a320 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7af00087d0 tx=0x7f7af0008c90 comp rx=0 tx=0).stop 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 shutdown_connections 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 --2- 172.21.15.153:0/3429870644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x55c2fbddd600 0x55c2fbe8a320 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 >> 172.21.15.153:0/3429870644 conn(0x55c2fbde5650 msgr2=0x55c2fbdddb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:21.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 shutdown_connections 2024-09-10T20:26:21.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 -- 172.21.15.153:0/3429870644 wait complete. 2024-09-10T20:26:21.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.390+0000 7f7b031ea100 1 librados: shutdown 2024-09-10T20:26:21.402 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-10T20:26:21.403 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-10T20:26:21.404 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-10T20:26:21.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient: get_monmap_and_config 2024-09-10T20:26:21.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient: build_initial_monmap 2024-09-10T20:26:21.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient: monmap: 2024-09-10T20:26:21.531 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:21.530533+0000 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:21.530533+0000 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:21.532 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 1 Processor -- start 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 1 -- start start 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient: init 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd31c0a41a0 0x7fd31c0a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient(hunting): picked mon.noname-a con 0x7fd31c0a41a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c0a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient(hunting): picked mon.noname-b con 0x7fd31c0a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:21.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd31c0a49a0 0x7fd31c0a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient(hunting): picked mon.noname-c con 0x7fd31c0a49a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fd31c0a2570 con 0x7fd31c0a41a0 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd31c070400 con 0x7fd31c0a49a0 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd31c0a6a20 con 0x7fd31c0a4580 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd322b88640 10 monclient(hunting): authenticate will time out at 941.670546s 2024-09-10T20:26:21.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd31c0a41a0 0x7fd31c0a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c0a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd31c0a41a0 0x7fd31c0a3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42168/0 (socket says 172.21.15.153:42168) 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 1 -- 172.21.15.153:0/3986897387 learned_addr learned my addr 172.21.15.153:0/3986897387 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 10 monclient(hunting): get_auth_request con 0x7fd31c0a41a0 auth_method 0 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient(hunting): get_auth_request con 0x7fd31c0a4580 auth_method 0 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:21.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient(hunting): handle_auth_done global_id 4196 payload 274 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 1 -- 172.21.15.153:0/3986897387 >> v2:172.21.15.176:3300/0 conn(0x7fd31c0a49a0 msgr2=0x7fd31c0a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd321125640 1 --2- 172.21.15.153:0/3986897387 >> v2:172.21.15.176:3300/0 conn(0x7fd31c0a49a0 0x7fd31c0a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 1 --2- 172.21.15.153:0/3986897387 >> v2:172.21.15.176:3300/0 conn(0x7fd31c0a49a0 0x7fd31c0a5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 1 -- 172.21.15.153:0/3986897387 >> v2:172.21.15.153:3300/0 conn(0x7fd31c0a41a0 msgr2=0x7fd31c0a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 1 --2- 172.21.15.153:0/3986897387 >> v2:172.21.15.153:3300/0 conn(0x7fd31c0a41a0 0x7fd31c0a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:21.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 1 monclient: found mon.noname-b 2024-09-10T20:26:21.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:21.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:21.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd31bfff640 1 -- 172.21.15.153:0/3986897387 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c0a3c80 con 0x7fd31c0a4580 2024-09-10T20:26:21.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.526+0000 7fd320924640 1 --2- 172.21.15.153:0/3986897387 >> v2:172.21.15.153:3300/0 conn(0x7fd31c0a41a0 0x7fd31c0a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31bfff640 1 -- 172.21.15.153:0/3986897387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 msgr2=0x7fd31c0a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31bfff640 1 --2- 172.21.15.153:0/3986897387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c0a59d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0a7aa0 tx=0x7fd31001c340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f285b18984827926 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:21.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 1 -- 172.21.15.153:0/3986897387 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100b0070 con 0x7fd31c0a4580 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 10 monclient: dump: 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:21.545 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 1 -- 172.21.15.153:0/3986897387 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3100ab600 con 0x7fd31c0a4580 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 1 -- 172.21.15.153:0/3986897387 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100ab9d0 con 0x7fd31c0a4580 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd31b7fe640 10 monclient: dump: 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.546 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd322b88640 5 monclient: authenticate success, global_id 4196 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd322b88640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:21.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.530+0000 7fd322b88640 4 set_mon_vals no callback set 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient: shutdown 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 -- 172.21.15.153:0/3986897387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 msgr2=0x7fd31c0a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 --2- 172.21.15.153:0/3986897387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c0a59d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd31c0a7aa0 tx=0x7fd31001c340 comp rx=0 tx=0).stop 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 -- 172.21.15.153:0/3986897387 shutdown_connections 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 --2- 172.21.15.153:0/3986897387 >> v2:172.21.15.176:3300/0 conn(0x7fd31c0a49a0 0x7fd31c0a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 --2- 172.21.15.153:0/3986897387 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c0a59d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 --2- 172.21.15.153:0/3986897387 >> v2:172.21.15.153:3300/0 conn(0x7fd31c0a41a0 0x7fd31c0a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 -- 172.21.15.153:0/3986897387 >> 172.21.15.153:0/3986897387 conn(0x7fd31c0ab7e0 msgr2=0x7fd31c0a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 -- 172.21.15.153:0/3986897387 shutdown_connections 2024-09-10T20:26:21.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 -- 172.21.15.153:0/3986897387 wait complete. 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient: build_initial_monmap 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient: monmap: 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:21.549 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 librados: starting msgr at 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 librados: starting objecter 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 Processor -- start 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 -- start start 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 librados: setting wanted keys 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 librados: calling monclient init 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient: init 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:21.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd31c0a41a0 0x7fd31c128ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient(hunting): picked mon.noname-a con 0x7fd31c0a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c126a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient(hunting): picked mon.noname-c con 0x7fd31c0a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd31c0a49a0 0x7fd31c13e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient(hunting): picked mon.noname-b con 0x7fd31c0a49a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c142200 con 0x7fd31c0a41a0 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd31c0a6860 con 0x7fd31c0a49a0 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd31c142380 con 0x7fd31c0a4580 2024-09-10T20:26:21.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd322b88640 10 monclient(hunting): authenticate will time out at 941.677822s 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c126a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd31c0a41a0 0x7fd31c128ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c126a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55480/0 (socket says 172.21.15.153:55480) 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd31c0a41a0 0x7fd31c128ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42178/0 (socket says 172.21.15.153:42178) 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 1 -- 172.21.15.153:0/2997576028 learned_addr learned my addr 172.21.15.153:0/2997576028 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient(hunting): get_auth_request con 0x7fd31c0a4580 auth_method 0 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 10 monclient(hunting): get_auth_request con 0x7fd31c0a41a0 auth_method 0 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:21.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient(hunting): handle_auth_done global_id 4199 payload 824 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 1 -- 172.21.15.153:0/2997576028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd31c0a49a0 msgr2=0x7fd31c13e930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 1 --2- 172.21.15.153:0/2997576028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd31c0a49a0 0x7fd31c13e930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 1 -- 172.21.15.153:0/2997576028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd31c0a41a0 msgr2=0x7fd31c128ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 1 --2- 172.21.15.153:0/2997576028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd31c0a41a0 0x7fd31c128ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 1 monclient: found mon.noname-c 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd31bfff640 1 -- 172.21.15.153:0/2997576028 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd31c129a30 con 0x7fd31c0a4580 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.534+0000 7fd320924640 1 --2- 172.21.15.153:0/2997576028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd31c0a41a0 0x7fd31c128ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd31bfff640 1 -- 172.21.15.153:0/2997576028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 msgr2=0x7fd31c126a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd31bfff640 1 --2- 172.21.15.153:0/2997576028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c126a50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd31001cbe0 tx=0x7fd3100190e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 client.?.objecter ms_handle_connect 0x7fd31c0a4580 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 1 -- 172.21.15.153:0/2997576028 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100b8020 con 0x7fd31c0a4580 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:21.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 monclient: dump: 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 monclient: _finish_auth 0 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 monclient: _check_auth_tickets 2024-09-10T20:26:21.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 1 -- 172.21.15.153:0/2997576028 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3100b4a20 con 0x7fd31c0a4580 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 5 monclient: authenticate success, global_id 4199 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 10 monclient: _renew_subs 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd31c0a21c0 con 0x7fd31c0a4580 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 1 -- 172.21.15.153:0/2997576028 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3100b0040 con 0x7fd31c0a4580 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 monclient: dump: 2024-09-10T20:26:21.556 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 10 client.4199.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 10 monclient: _renew_subs 2024-09-10T20:26:21.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd31c12a2d0 con 0x7fd31c0a4580 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 1 librados: init done 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd31a7fc640 4 set_mon_vals no callback set 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd31c0a3720 con 0x7fd31c0a4580 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 1 -- 172.21.15.153:0/2997576028 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fd3100bb020 con 0x7fd31c0a4580 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 1 --2- 172.21.15.153:0/2997576028 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd30803cef0 0x7fd30803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 1 -- 172.21.15.153:0/2997576028 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7fd3100e9240 con 0x7fd31c0a4580 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 10 client.4199.objecter ms_dispatch 0x7fd31c000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 3 client.4199.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:21.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 3 client.4199.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:21.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd3197fa640 20 client.4199.objecter dump_active .. 0 homeless 2024-09-10T20:26:21.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd320924640 1 --2- 172.21.15.153:0/2997576028 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd30803cef0 0x7fd30803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.538+0000 7fd320924640 10 monclient: get_auth_request con 0x7fd30803cef0 auth_method 0 2024-09-10T20:26:21.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.542+0000 7fd320924640 1 --2- 172.21.15.153:0/2997576028 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd30803cef0 0x7fd30803f3b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009630 tx=0x7fd30c009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:21.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.542+0000 7fd3197fa640 10 client.4199.objecter ms_handle_connect 0x7fd30803cef0 2024-09-10T20:26:21.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.542+0000 7fd3197fa640 1 -- 172.21.15.153:0/2997576028 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fd3100c2030 con 0x7fd31c0a4580 2024-09-10T20:26:21.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.542+0000 7fd3197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:21.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.542+0000 7fd3197fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:21.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.710+0000 7fd322b88640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-10T20:26:21.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.710+0000 7fd322b88640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-10T20:26:21.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.710+0000 7fd322b88640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:21.717 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.710+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fd31c06cd40 con 0x7fd31c0a4580 2024-09-10T20:26:21.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.710+0000 7fd3197fa640 1 -- 172.21.15.153:0/2997576028 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v4) ==== 86+0+159 (secure 0 0 0) 0x7fd31c129a30 con 0x7fd31c0a4580 2024-09-10T20:26:21.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.710+0000 7fd3197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-10T20:26:21.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.710+0000 7fd3197fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:21.718 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:21.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 10 librados: watch_flush enter 2024-09-10T20:26:21.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 10 librados: watch_flush exit 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 20 client.4199.objecter shutdown clearing up homeless session... 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 10 client.4199.objecter successfully canceled tick 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd30803cef0 msgr2=0x7fd30803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 --2- 172.21.15.153:0/2997576028 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd30803cef0 0x7fd30803f3b0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fd30c009630 tx=0x7fd30c009240 comp rx=0 tx=0).stop 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 10 monclient: shutdown 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 msgr2=0x7fd31c126a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 --2- 172.21.15.153:0/2997576028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c126a50 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd31001cbe0 tx=0x7fd3100190e0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 shutdown_connections 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 --2- 172.21.15.153:0/2997576028 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd31c0a49a0 0x7fd31c13e930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 --2- 172.21.15.153:0/2997576028 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd30803cef0 0x7fd30803f3b0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 --2- 172.21.15.153:0/2997576028 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd31c0a4580 0x7fd31c126a50 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 --2- 172.21.15.153:0/2997576028 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd31c0a41a0 0x7fd31c128ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 >> 172.21.15.153:0/2997576028 conn(0x7fd31c0ab7e0 msgr2=0x7fd31c0a0080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 shutdown_connections 2024-09-10T20:26:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 -- 172.21.15.153:0/2997576028 wait complete. 2024-09-10T20:26:21.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.714+0000 7fd322b88640 1 librados: shutdown 2024-09-10T20:26:21.736 INFO:teuthology.orchestra.run.smithi153.stdout:[{"version":4,"timestamp":"2024-09-10T20:26:11.314839+0000","name":"","changes":[{"name":"osd.0/osd_mclock_max_capacity_iops_ssd","new_value":"4997.963447"}]}] 2024-09-10T20:26:21.736 INFO:tasks.ceph_manager:config epoch is 4 2024-09-10T20:26:21.736 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-10T20:26:21.736 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-10T20:26:21.737 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-10T20:26:21.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.846+0000 7f6467473640 10 monclient: get_monmap_and_config 2024-09-10T20:26:21.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.846+0000 7f6467473640 10 monclient: build_initial_monmap 2024-09-10T20:26:21.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.846+0000 7f6467473640 10 monclient: monmap: 2024-09-10T20:26:21.854 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:21.854 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.854 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:21.852735+0000 2024-09-10T20:26:21.855 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:21.852735+0000 2024-09-10T20:26:21.855 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:21.855 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.855 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:21.855 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:21.855 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:21.855 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:21.855 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:21.856 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.846+0000 7f6467473640 1 Processor -- start 2024-09-10T20:26:21.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.846+0000 7f6467473640 1 -- start start 2024-09-10T20:26:21.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.846+0000 7f6467473640 10 monclient: init 2024-09-10T20:26:21.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:21.858 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 10 monclient: _add_conns ranks=[0,3,1,2,4] 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f64601364f0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f64601360a0 0x7f646013b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 10 monclient(hunting): picked mon.noname-a con 0x7f64601360a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f64600a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64600a27a0 con 0x7f64601364f0 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64600a2a80 con 0x7f64600a1990 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f64600a2550 con 0x7f64601360a0 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6467473640 10 monclient(hunting): authenticate will time out at 941.992672s 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f64601360a0 0x7f646013b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f646520f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f64601364f0 0x7f64601368d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60422/0 (socket says 172.21.15.153:60422) 2024-09-10T20:26:21.859 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f646520f640 1 -- 172.21.15.153:0/971592536 learned_addr learned my addr 172.21.15.153:0/971592536 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient(hunting): get_auth_request con 0x7f64601360a0 auth_method 0 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient(hunting): handle_auth_done global_id 4218 payload 274 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 1 -- 172.21.15.153:0/971592536 >> v1:172.21.15.176:6789/0 conn(0x7f64600a1990 legacy=0x7f646013d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 1 -- 172.21.15.153:0/971592536 >> v1:172.21.15.153:6789/0 conn(0x7f64601364f0 legacy=0x7f64601368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient: _finish_hunting 0 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 1 monclient: found mon.noname-a 2024-09-10T20:26:21.860 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:21.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:21.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.850+0000 7f6464a0e640 1 -- 172.21.15.153:0/971592536 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64600a2d60 con 0x7f64601360a0 2024-09-10T20:26:21.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6464a0e640 1 -- 172.21.15.153:0/971592536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601360a0 msgr2=0x7f646013b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6464a0e640 1 --2- 172.21.15.153:0/971592536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601360a0 0x7f646013b420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f644c0097c0 tx=0x7f644c01c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac00f6ef36a66bb5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 1 -- 172.21.15.153:0/971592536 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c0b0070 con 0x7f64601360a0 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 10 monclient: dump: 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.866 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 10 monclient: _finish_auth 0 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 1 -- 172.21.15.153:0/971592536 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f644c0ab600 con 0x7f64601360a0 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 5 monclient: authenticate success, global_id 4218 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 4 set_mon_vals no callback set 2024-09-10T20:26:21.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 1 -- 172.21.15.153:0/971592536 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f644c0ab9d0 con 0x7f64601360a0 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f645ffff640 10 monclient: dump: 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.868 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:21.869 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 10 monclient: shutdown 2024-09-10T20:26:21.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:21.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 1 -- 172.21.15.153:0/971592536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601360a0 msgr2=0x7f646013b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 1 --2- 172.21.15.153:0/971592536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601360a0 0x7f646013b420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f644c0097c0 tx=0x7f644c01c360 comp rx=0 tx=0).stop 2024-09-10T20:26:21.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 1 -- 172.21.15.153:0/971592536 shutdown_connections 2024-09-10T20:26:21.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 1 --2- 172.21.15.153:0/971592536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64601360a0 0x7f646013b420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 1 -- 172.21.15.153:0/971592536 >> 172.21.15.153:0/971592536 conn(0x7f64600a71b0 msgr2=0x7f64600a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:21.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 1 -- 172.21.15.153:0/971592536 shutdown_connections 2024-09-10T20:26:21.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.854+0000 7f6467473640 1 -- 172.21.15.153:0/971592536 wait complete. 2024-09-10T20:26:21.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient: build_initial_monmap 2024-09-10T20:26:21.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient: monmap: 2024-09-10T20:26:21.871 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:21.872 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:21.872 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:21.872 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:21.872 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:21.872 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.872 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:21.872 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:21.873 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:21.873 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 librados: starting msgr at 2024-09-10T20:26:21.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 librados: starting objecter 2024-09-10T20:26:21.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 Processor -- start 2024-09-10T20:26:21.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 -- start start 2024-09-10T20:26:21.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 librados: setting wanted keys 2024-09-10T20:26:21.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 librados: calling monclient init 2024-09-10T20:26:21.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient: init 2024-09-10T20:26:21.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:21.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:26:21.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64600a1990 0x7f64601d06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient(hunting): picked mon.noname-a con 0x7f64600a1990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:21.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64601360a0 0x7f646013e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient(hunting): picked mon.noname-c con 0x7f64601360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:21.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64601364f0 0x7f646013e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient(hunting): picked mon.noname-b con 0x7f64601364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:21.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6460141d80 con 0x7f64600a1990 2024-09-10T20:26:21.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6460141c00 con 0x7f64601364f0 2024-09-10T20:26:21.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:21.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6460141f00 con 0x7f64601360a0 2024-09-10T20:26:21.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:21.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6467473640 10 monclient(hunting): authenticate will time out at 942.001755s 2024-09-10T20:26:21.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64600a1990 0x7f64601d06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6464a0e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64601360a0 0x7f646013e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64600a1990 0x7f64601d06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42192/0 (socket says 172.21.15.153:42192) 2024-09-10T20:26:21.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 1 -- 172.21.15.153:0/3019622980 learned_addr learned my addr 172.21.15.153:0/3019622980 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:21.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient(hunting): get_auth_request con 0x7f64600a1990 auth_method 0 2024-09-10T20:26:21.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:21.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:21.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:21.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6464a0e640 10 monclient(hunting): get_auth_request con 0x7f64601360a0 auth_method 0 2024-09-10T20:26:21.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6464a0e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:21.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6464a0e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:21.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6464a0e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:21.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:21.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:21.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:21.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6464a0e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:21.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6464a0e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:21.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6464a0e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:21.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6465a10640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64601364f0 0x7f646013e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient(hunting): handle_auth_done global_id 4221 payload 824 2024-09-10T20:26:21.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 1 -- 172.21.15.153:0/3019622980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64601360a0 msgr2=0x7f646013e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64601360a0 0x7f646013e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 1 -- 172.21.15.153:0/3019622980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64601364f0 msgr2=0x7f646013e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:21.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64601364f0 0x7f646013e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:21.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient: _finish_hunting 0 2024-09-10T20:26:21.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 1 monclient: found mon.noname-a 2024-09-10T20:26:21.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:21.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:21.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f646520f640 1 -- 172.21.15.153:0/3019622980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f646013bd20 con 0x7f64600a1990 2024-09-10T20:26:21.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6465a10640 10 monclient: get_auth_request con 0x7f64601364f0 auth_method 0 2024-09-10T20:26:21.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6464a0e640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64601360a0 0x7f646013e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:21.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.858+0000 7f6465a10640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64601364f0 0x7f646013e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:21.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f646520f640 1 -- 172.21.15.153:0/3019622980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64600a1990 msgr2=0x7f64601d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:21.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f646520f640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64600a1990 0x7f64601d06d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f645000cc10 tx=0x7f645000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 client.?.objecter ms_handle_connect 0x7f64600a1990 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 1 -- 172.21.15.153:0/3019622980 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6450012400 con 0x7f64600a1990 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 monclient: dump: 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.888 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 1 -- 172.21.15.153:0/3019622980 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64500125e0 con 0x7f64600a1990 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 5 monclient: authenticate success, global_id 4221 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 10 monclient: _renew_subs 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f646013b740 con 0x7f64600a1990 2024-09-10T20:26:21.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 1 -- 172.21.15.153:0/3019622980 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64500129d0 con 0x7f64600a1990 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 monclient: dump: 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:21.890 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645f7fe640 4 set_mon_vals no callback set 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 10 monclient: _renew_subs 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f646013c6a0 con 0x7f64600a1990 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 1 librados: init done 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64601368d0 con 0x7f64600a1990 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 1 -- 172.21.15.153:0/3019622980 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f6450012bf0 con 0x7f64600a1990 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 1 --2- 172.21.15.153:0/3019622980 >> v2:172.21.15.176:6812/3003473398 conn(0x7f644803cea0 0x7f644803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:21.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 1 -- 172.21.15.153:0/3019622980 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f645004fa00 con 0x7f64600a1990 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 10 client.4221.objecter ms_dispatch 0x7f6460000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 3 client.4221.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 3 client.4221.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f645dffb640 20 client.4221.objecter dump_active .. 0 homeless 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6464a0e640 1 --2- 172.21.15.153:0/3019622980 >> v2:172.21.15.176:6812/3003473398 conn(0x7f644803cea0 0x7f644803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.862+0000 7f6464a0e640 10 monclient: get_auth_request con 0x7f644803cea0 auth_method 0 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.866+0000 7f645dffb640 10 client.4221.objecter ms_handle_connect 0x7f644803cea0 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.866+0000 7f6464a0e640 1 --2- 172.21.15.153:0/3019622980 >> v2:172.21.15.176:6812/3003473398 conn(0x7f644803cea0 0x7f644803f360 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f64600a12f0 tx=0x7f644c0a4f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.866+0000 7f645dffb640 1 -- 172.21.15.153:0/3019622980 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f645001bc00 con 0x7f64600a1990 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.866+0000 7f645dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:21.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:21.866+0000 7f645dffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:22.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.082+0000 7f6467473640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-10T20:26:22.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.082+0000 7f6467473640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-10T20:26:22.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.082+0000 7f6467473640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.082+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7f646006a640 con 0x7f64600a1990 2024-09-10T20:26:22.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f645dffb640 1 -- 172.21.15.153:0/3019622980 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+93154 (secure 0 0 0) 0x7f645001a360 con 0x7f64600a1990 2024-09-10T20:26:22.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f645dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-10T20:26:22.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f645dffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:22.092 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:22.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 10 librados: watch_flush enter 2024-09-10T20:26:22.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 10 librados: watch_flush exit 2024-09-10T20:26:22.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 20 client.4221.objecter shutdown clearing up homeless session... 2024-09-10T20:26:22.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 10 client.4221.objecter successfully canceled tick 2024-09-10T20:26:22.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 >> v2:172.21.15.176:6812/3003473398 conn(0x7f644803cea0 msgr2=0x7f644803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 --2- 172.21.15.153:0/3019622980 >> v2:172.21.15.176:6812/3003473398 conn(0x7f644803cea0 0x7f644803f360 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f64600a12f0 tx=0x7f644c0a4f90 comp rx=0 tx=0).stop 2024-09-10T20:26:22.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 10 monclient: shutdown 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64600a1990 msgr2=0x7f64601d06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64600a1990 0x7f64601d06d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f645000cc10 tx=0x7f645000a430 comp rx=0 tx=0).stop 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 shutdown_connections 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f64601364f0 0x7f646013e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 --2- 172.21.15.153:0/3019622980 >> v2:172.21.15.176:6812/3003473398 conn(0x7f644803cea0 0x7f644803f360 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f64601360a0 0x7f646013e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 --2- 172.21.15.153:0/3019622980 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f64600a1990 0x7f64601d06d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 >> 172.21.15.153:0/3019622980 conn(0x7f64600a71b0 msgr2=0x7f64600a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 shutdown_connections 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.086+0000 7f6467473640 1 -- 172.21.15.153:0/3019622980 wait complete. 2024-09-10T20:26:22.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.090+0000 7f6467473640 1 librados: shutdown 2024-09-10T20:26:22.110 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":5,"flags":0,"active_gid":4106,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6812","nonce":3003473398}]},"active_addr":"172.21.15.176:6812/3003473398","active_change":"2024-09-10T20:26:05.055337+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":2602544874}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":4103688769}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":3051681351}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.176:0","nonce":298535533}]}]} 2024-09-10T20:26:22.114 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-10T20:26:22.115 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-10T20:26:22.115 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:26:22.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient: get_monmap_and_config 2024-09-10T20:26:22.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient: monmap: 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:22.231042+0000 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:22.231042+0000 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:22.232 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 1 Processor -- start 2024-09-10T20:26:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 1 -- start start 2024-09-10T20:26:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient: init 2024-09-10T20:26:22.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient: _add_conns ranks=[1,0,4,2,3] 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f10d40a28c0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f10d40a3060 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f10d413b7d0 0x7f10d413dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient(hunting): picked mon.noname-c con 0x7f10d413b7d0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10d409f890 con 0x7f10d40a3060 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10d409de50 con 0x7f10d40a28c0 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f10d40a34e0 con 0x7f10d413b7d0 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10db1c4640 10 monclient(hunting): authenticate will time out at 942.370840s 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d3fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f10d40a3060 0x7f10d409d740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60432/0 (socket says 172.21.15.153:60432) 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d3fff640 1 -- 172.21.15.153:0/1124454972 learned_addr learned my addr 172.21.15.153:0/1124454972 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:22.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 1 --2- 172.21.15.153:0/1124454972 >> v2:172.21.15.176:3300/0 conn(0x7f10d413b7d0 0x7f10d413dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient(hunting): get_auth_request con 0x7f10d413b7d0 auth_method 0 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient(hunting): handle_auth_done global_id 4195 payload 274 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 1 -- 172.21.15.153:0/1124454972 >> v1:172.21.15.176:6789/0 conn(0x7f10d40a28c0 legacy=0x7f10d409d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 1 -- 172.21.15.153:0/1124454972 >> v1:172.21.15.153:6789/0 conn(0x7f10d40a3060 legacy=0x7f10d409d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 1 monclient: found mon.noname-c 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:26:22.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.226+0000 7f10d9761640 1 -- 172.21.15.153:0/1124454972 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d409e350 con 0x7f10d413b7d0 2024-09-10T20:26:22.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 1 -- 172.21.15.153:0/1124454972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d413b7d0 msgr2=0x7f10d413dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 1 --2- 172.21.15.153:0/1124454972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d413b7d0 0x7f10d413dbc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009d90 tx=0x7f10cc01c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f5490600c5320d07 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 1 -- 172.21.15.153:0/1124454972 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc0b2070 con 0x7f10d413b7d0 2024-09-10T20:26:22.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:22.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 10 monclient: dump: 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 1 -- 172.21.15.153:0/1124454972 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10cc0a5800 con 0x7f10d413b7d0 2024-09-10T20:26:22.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 1 -- 172.21.15.153:0/1124454972 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10cc0a5bd0 con 0x7f10d413b7d0 2024-09-10T20:26:22.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:22.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d37fe640 10 monclient: dump: 2024-09-10T20:26:22.241 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.241 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.241 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.241 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 5 monclient: authenticate success, global_id 4195 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 4 set_mon_vals no callback set 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient: shutdown 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 -- 172.21.15.153:0/1124454972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d413b7d0 msgr2=0x7f10d413dbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 --2- 172.21.15.153:0/1124454972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d413b7d0 0x7f10d413dbc0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f10cc009d90 tx=0x7f10cc01c830 comp rx=0 tx=0).stop 2024-09-10T20:26:22.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 -- 172.21.15.153:0/1124454972 shutdown_connections 2024-09-10T20:26:22.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 --2- 172.21.15.153:0/1124454972 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d413b7d0 0x7f10d413dbc0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 -- 172.21.15.153:0/1124454972 >> 172.21.15.153:0/1124454972 conn(0x7f10d40a71b0 msgr2=0x7f10d40a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:22.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 -- 172.21.15.153:0/1124454972 shutdown_connections 2024-09-10T20:26:22.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 -- 172.21.15.153:0/1124454972 wait complete. 2024-09-10T20:26:22.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient: monmap: 2024-09-10T20:26:22.247 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.247 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:22.248 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:22.248 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:22.248 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.248 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.248 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:22.248 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:22.248 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 librados: starting msgr at 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 librados: starting objecter 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 Processor -- start 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 -- start start 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 librados: setting wanted keys 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 librados: calling monclient init 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient: init 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d40a28c0 0x7f10d4134f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient(hunting): picked mon.noname-a con 0x7f10d40a28c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d40a3060 0x7f10d41354a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient(hunting): picked mon.noname-b con 0x7f10d40a3060 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d413b7d0 0x7f10d4071670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient(hunting): picked mon.noname-c con 0x7f10d413b7d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10d4140cb0 con 0x7f10d40a28c0 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f10d4140b30 con 0x7f10d40a3060 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f10d4140e30 con 0x7f10d413b7d0 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10db1c4640 10 monclient(hunting): authenticate will time out at 942.378766s 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d413b7d0 0x7f10d4071670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d40a28c0 0x7f10d4134f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d413b7d0 0x7f10d4071670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55510/0 (socket says 172.21.15.153:55510) 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 1 -- 172.21.15.153:0/1409311374 learned_addr learned my addr 172.21.15.153:0/1409311374 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 10 monclient(hunting): get_auth_request con 0x7f10d413b7d0 auth_method 0 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient(hunting): get_auth_request con 0x7f10d40a28c0 auth_method 0 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d3fff640 1 --2- 172.21.15.153:0/1409311374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d40a3060 0x7f10d41354a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d3fff640 10 monclient(hunting): get_auth_request con 0x7f10d40a3060 auth_method 0 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 1 -- 172.21.15.153:0/1409311374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d413b7d0 msgr2=0x7f10d4071670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 1 --2- 172.21.15.153:0/1409311374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d413b7d0 0x7f10d4071670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 1 -- 172.21.15.153:0/1409311374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d40a3060 msgr2=0x7f10d41354a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 1 --2- 172.21.15.153:0/1409311374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d40a3060 0x7f10d41354a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 1 monclient: found mon.noname-a 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d8f60640 1 -- 172.21.15.153:0/1409311374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10d411db60 con 0x7f10d40a28c0 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.234+0000 7f10d9761640 1 --2- 172.21.15.153:0/1409311374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d413b7d0 0x7f10d4071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d8f60640 1 -- 172.21.15.153:0/1409311374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d40a28c0 msgr2=0x7f10d4134f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d8f60640 1 --2- 172.21.15.153:0/1409311374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d40a28c0 0x7f10d4134f60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f10c400b410 tx=0x7f10c400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 client.?.objecter ms_handle_connect 0x7f10d40a28c0 2024-09-10T20:26:22.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 1 -- 172.21.15.153:0/1409311374 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4015030 con 0x7f10d40a28c0 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 monclient: dump: 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.254 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 1 -- 172.21.15.153:0/1409311374 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10c4004020 con 0x7f10d40a28c0 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 5 monclient: authenticate success, global_id 4224 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 10 monclient: _renew_subs 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10d411de40 con 0x7f10d40a28c0 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 1 -- 172.21.15.153:0/1409311374 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10c4004410 con 0x7f10d40a28c0 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:22.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 monclient: dump: 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 10 monclient: _renew_subs 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10d411d880 con 0x7f10d40a28c0 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d27fc640 4 set_mon_vals no callback set 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 1 librados: init done 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10d409d030 con 0x7f10d40a28c0 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 1 -- 172.21.15.153:0/1409311374 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f10c4018070 con 0x7f10d40a28c0 2024-09-10T20:26:22.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 1 --2- 172.21.15.153:0/1409311374 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10c003cf00 0x7f10c003f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 1 -- 172.21.15.153:0/1409311374 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f10c404ed20 con 0x7f10d40a28c0 2024-09-10T20:26:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 10 client.4224.objecter ms_dispatch 0x7f10d4000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 3 client.4224.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 3 client.4224.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d17fa640 20 client.4224.objecter dump_active .. 0 homeless 2024-09-10T20:26:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d3fff640 1 --2- 172.21.15.153:0/1409311374 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10c003cf00 0x7f10c003f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.238+0000 7f10d3fff640 10 monclient: get_auth_request con 0x7f10c003cf00 auth_method 0 2024-09-10T20:26:22.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.242+0000 7f10d17fa640 10 client.4224.objecter ms_handle_connect 0x7f10c003cf00 2024-09-10T20:26:22.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.242+0000 7f10d3fff640 1 --2- 172.21.15.153:0/1409311374 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10c003cf00 0x7f10c003f3c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f10c8007fd0 tx=0x7f10c8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.242+0000 7f10d17fa640 1 -- 172.21.15.153:0/1409311374 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f10c4010380 con 0x7f10d40a28c0 2024-09-10T20:26:22.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.242+0000 7f10d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.242+0000 7f10d17fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:22.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.414+0000 7f10db1c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:22.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.414+0000 7f10db1c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:22.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.414+0000 7f10db1c4640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.414+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f10d406a640 con 0x7f10d40a28c0 2024-09-10T20:26:22.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.414+0000 7f10d17fa640 1 -- 172.21.15.153:0/1409311374 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10987 (secure 0 0 0) 0x7f10c4003dc0 con 0x7f10d40a28c0 2024-09-10T20:26:22.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.414+0000 7f10d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:22.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.414+0000 7f10d17fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:22.420 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:22.420 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":20,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","created":"2024-09-10T20:25:59.587062+0000","modified":"2024-09-10T20:26:21.366705+0000","last_up_change":"2024-09-10T20:26:12.282988+0000","last_in_change":"2024-09-10T20:26:02.414479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:26:13.125072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-10T20:26:18.322750+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"faf817a9-b230-4970-b8c5-e5637fd29193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4216589932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":4216589932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":4216589932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4216589932}]},"public_addr":"172.21.15.153:6800/4216589932","cluster_addr":"172.21.15.153:6801/4216589932","heartbeat_back_addr":"172.21.15.153:6803/4216589932","heartbeat_front_addr":"172.21.15.153:6802/4216589932","state":["exists","up"]},{"osd":1,"uuid":"f85962f8-8bd9-4e26-8d66-49b6528617a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1715391555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":1715391555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1715391555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1715391555}]},"public_addr":"172.21.15.153:6804/1715391555","cluster_addr":"172.21.15.153:6805/1715391555","heartbeat_back_addr":"172.21.15.153:6807/1715391555","heartbeat_front_addr":"172.21.15.153:6806/1715391555","state":["exists","up"]},{"osd":2,"uuid":"18d54b55-1393-42b9-9fcc-84202659cf31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1626005750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":1626005750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":1626005750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1626005750}]},"public_addr":"172.21.15.153:6808/1626005750","cluster_addr":"172.21.15.153:6809/1626005750","heartbeat_back_addr":"172.21.15.153:6811/1626005750","heartbeat_front_addr":"172.21.15.153:6810/1626005750","state":["exists","up"]},{"osd":3,"uuid":"5149b9c4-35d7-4ffb-9fac-7b3d9b660101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2953083416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":2953083416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6811","nonce":2953083416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2953083416}]},"public_addr":"172.21.15.176:6808/2953083416","cluster_addr":"172.21.15.176:6809/2953083416","heartbeat_back_addr":"172.21.15.176:6811/2953083416","heartbeat_front_addr":"172.21.15.176:6810/2953083416","state":["exists","up"]},{"osd":4,"uuid":"61e0c0cc-4301-4947-9f1d-838d2d7bbfea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4030285519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6805","nonce":4030285519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6807","nonce":4030285519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4030285519}]},"public_addr":"172.21.15.176:6804/4030285519","cluster_addr":"172.21.15.176:6805/4030285519","heartbeat_back_addr":"172.21.15.176:6807/4030285519","heartbeat_front_addr":"172.21.15.176:6806/4030285519","state":["exists","up"]},{"osd":5,"uuid":"2feacd82-c45a-4d90-8210-ff9faecc2d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":2372911183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6801","nonce":2372911183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6803","nonce":2372911183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2372911183}]},"public_addr":"172.21.15.176:6800/2372911183","cluster_addr":"172.21.15.176:6801/2372911183","heartbeat_back_addr":"172.21.15.176:6803/2372911183","heartbeat_front_addr":"172.21.15.176:6802/2372911183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.301670+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:10.037323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.733818+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364748+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.173071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:26:22.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 10 librados: watch_flush enter 2024-09-10T20:26:22.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 10 librados: watch_flush exit 2024-09-10T20:26:22.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 20 client.4224.objecter shutdown clearing up homeless session... 2024-09-10T20:26:22.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 10 client.4224.objecter successfully canceled tick 2024-09-10T20:26:22.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10c003cf00 msgr2=0x7f10c003f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 --2- 172.21.15.153:0/1409311374 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10c003cf00 0x7f10c003f3c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f10c8007fd0 tx=0x7f10c8009040 comp rx=0 tx=0).stop 2024-09-10T20:26:22.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 10 monclient: shutdown 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d40a28c0 msgr2=0x7f10d4134f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 --2- 172.21.15.153:0/1409311374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d40a28c0 0x7f10d4134f60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f10c400b410 tx=0x7f10c400b8d0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 shutdown_connections 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 --2- 172.21.15.153:0/1409311374 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10c003cf00 0x7f10c003f3c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 --2- 172.21.15.153:0/1409311374 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d413b7d0 0x7f10d4071670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 --2- 172.21.15.153:0/1409311374 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10d40a3060 0x7f10d41354a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 --2- 172.21.15.153:0/1409311374 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10d40a28c0 0x7f10d4134f60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 >> 172.21.15.153:0/1409311374 conn(0x7f10d40a71b0 msgr2=0x7f10d413c790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 shutdown_connections 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 -- 172.21.15.153:0/1409311374 wait complete. 2024-09-10T20:26:22.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.418+0000 7f10db1c4640 1 librados: shutdown 2024-09-10T20:26:22.438 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-10T20:26:22.438 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:26:22.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.546+0000 7f0d7ffe7640 10 monclient: get_monmap_and_config 2024-09-10T20:26:22.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.546+0000 7f0d7ffe7640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.546+0000 7f0d7ffe7640 10 monclient: monmap: 2024-09-10T20:26:22.554 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.554 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.554 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:22.552980+0000 2024-09-10T20:26:22.554 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:22.552980+0000 2024-09-10T20:26:22.555 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.555 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.555 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:22.555 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:22.555 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:22.555 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:22.555 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:22.556 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.546+0000 7f0d7ffe7640 1 Processor -- start 2024-09-10T20:26:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.546+0000 7f0d7ffe7640 1 -- start start 2024-09-10T20:26:22.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.546+0000 7f0d7ffe7640 10 monclient: init 2024-09-10T20:26:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-10T20:26:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0d780a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a4580 0x7f0d7813d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 10 monclient(hunting): picked mon.noname-b con 0x7f0d780a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0d780a49a0 0x7f0d7813fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 10 monclient(hunting): picked mon.noname-c con 0x7f0d780a49a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d780a6a20 con 0x7f0d780a41a0 2024-09-10T20:26:22.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0d780a2570 con 0x7f0d780a49a0 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0d78070400 con 0x7f0d780a4580 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7ffe7640 10 monclient(hunting): authenticate will time out at 942.692629s 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a4580 0x7f0d7813d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a4580 0x7f0d7813d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55512/0 (socket says 172.21.15.153:55512) 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 1 -- 172.21.15.153:0/1106095075 learned_addr learned my addr 172.21.15.153:0/1106095075 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient(hunting): get_auth_request con 0x7f0d780a4580 auth_method 0 2024-09-10T20:26:22.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7e584640 1 --2- 172.21.15.153:0/1106095075 >> v2:172.21.15.176:3300/0 conn(0x7f0d780a49a0 0x7f0d7813fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7e584640 10 monclient(hunting): get_auth_request con 0x7f0d780a49a0 auth_method 0 2024-09-10T20:26:22.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7e584640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7e584640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7e584640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient(hunting): handle_auth_done global_id 4208 payload 274 2024-09-10T20:26:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 1 -- 172.21.15.153:0/1106095075 >> v2:172.21.15.176:3300/0 conn(0x7f0d780a49a0 msgr2=0x7f0d7813fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 1 --2- 172.21.15.153:0/1106095075 >> v2:172.21.15.176:3300/0 conn(0x7f0d780a49a0 0x7f0d7813fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 1 -- 172.21.15.153:0/1106095075 >> v1:172.21.15.176:6789/0 conn(0x7f0d780a41a0 legacy=0x7f0d780a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 1 monclient: found mon.noname-b 2024-09-10T20:26:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.550+0000 7f0d7d582640 1 -- 172.21.15.153:0/1106095075 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d780a85a0 con 0x7f0d780a4580 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7d582640 1 -- 172.21.15.153:0/1106095075 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a4580 msgr2=0x7f0d7813d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7d582640 1 --2- 172.21.15.153:0/1106095075 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a4580 0x7f0d7813d7c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d74009870 tx=0x7f0d7401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaf14206de89678f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 1 -- 172.21.15.153:0/1106095075 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d740b0070 con 0x7f0d780a4580 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 10 monclient: dump: 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.569 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 5 monclient: authenticate success, global_id 4208 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 1 -- 172.21.15.153:0/1106095075 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d740a9710 con 0x7f0d780a4580 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 4 set_mon_vals no callback set 2024-09-10T20:26:22.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 1 -- 172.21.15.153:0/1106095075 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d740a9ae0 con 0x7f0d780a4580 2024-09-10T20:26:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7cd81640 10 monclient: dump: 2024-09-10T20:26:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.571 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient: shutdown 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/1106095075 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a4580 msgr2=0x7f0d7813d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 --2- 172.21.15.153:0/1106095075 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a4580 0x7f0d7813d7c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0d74009870 tx=0x7f0d7401c420 comp rx=0 tx=0).stop 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/1106095075 shutdown_connections 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 --2- 172.21.15.153:0/1106095075 >> v2:172.21.15.176:3300/0 conn(0x7f0d780a49a0 0x7f0d7813fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 --2- 172.21.15.153:0/1106095075 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a4580 0x7f0d7813d7c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/1106095075 >> 172.21.15.153:0/1106095075 conn(0x7f0d780ab7e0 msgr2=0x7f0d780a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/1106095075 shutdown_connections 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/1106095075 wait complete. 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient: monmap: 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.573 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 librados: starting msgr at 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 librados: starting objecter 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 Processor -- start 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 -- start start 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 librados: setting wanted keys 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 librados: calling monclient init 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient: init 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d780a41a0 0x7f0d781219f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient(hunting): picked mon.noname-b con 0x7f0d780a41a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d780a4580 0x7f0d7809edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient(hunting): picked mon.noname-a con 0x7f0d780a4580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a49a0 0x7f0d78138860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient(hunting): picked mon.noname-c con 0x7f0d780a49a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0d78142e00 con 0x7f0d780a4580 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0d78142c80 con 0x7f0d780a41a0 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0d78142f80 con 0x7f0d780a49a0 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7ffe7640 10 monclient(hunting): authenticate will time out at 942.699716s 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7e584640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a49a0 0x7f0d78138860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7d582640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d780a4580 0x7f0d7809edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7e584640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a49a0 0x7f0d78138860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55516/0 (socket says 172.21.15.153:55516) 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7d582640 10 monclient(hunting): get_auth_request con 0x7f0d780a4580 auth_method 0 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7d582640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7d582640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7d582640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7e584640 1 -- 172.21.15.153:0/407237232 learned_addr learned my addr 172.21.15.153:0/407237232 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7e584640 10 monclient(hunting): get_auth_request con 0x7f0d780a49a0 auth_method 0 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7e584640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7e584640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.554+0000 7f0d7e584640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7e584640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7e584640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7e584640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7dd83640 1 --2- 172.21.15.153:0/407237232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d780a41a0 0x7f0d781219f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7dd83640 10 monclient(hunting): get_auth_request con 0x7f0d780a41a0 auth_method 0 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7dd83640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7dd83640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7dd83640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 10 monclient(hunting): handle_auth_done global_id 4227 payload 824 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 1 -- 172.21.15.153:0/407237232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a49a0 msgr2=0x7f0d78138860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 1 --2- 172.21.15.153:0/407237232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a49a0 0x7f0d78138860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 1 -- 172.21.15.153:0/407237232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d780a41a0 msgr2=0x7f0d781219f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 1 --2- 172.21.15.153:0/407237232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d780a41a0 0x7f0d781219f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 1 monclient: found mon.noname-a 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 1 -- 172.21.15.153:0/407237232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d780a21c0 con 0x7f0d780a4580 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7e584640 1 --2- 172.21.15.153:0/407237232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a49a0 0x7f0d78138860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 1 -- 172.21.15.153:0/407237232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d780a4580 msgr2=0x7f0d7809edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7d582640 1 --2- 172.21.15.153:0/407237232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d780a4580 0x7f0d7809edf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0d740ab000 tx=0x7f0d740191c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 client.?.objecter ms_handle_connect 0x7f0d780a4580 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:22.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 1 -- 172.21.15.153:0/407237232 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d740b8020 con 0x7f0d780a4580 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 monclient: dump: 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 1 -- 172.21.15.153:0/407237232 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d740ab940 con 0x7f0d780a4580 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 5 monclient: authenticate success, global_id 4227 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 10 monclient: _renew_subs 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d780a03d0 con 0x7f0d780a4580 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 1 -- 172.21.15.153:0/407237232 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d740b0040 con 0x7f0d780a4580 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 10 monclient: dump: 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.580 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.581 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 10 monclient: _renew_subs 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6ffff640 4 set_mon_vals no callback set 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d780a0b60 con 0x7f0d780a4580 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 1 librados: init done 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d780a3720 con 0x7f0d780a4580 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.558+0000 7f0d6e7fc640 1 -- 172.21.15.153:0/407237232 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f0d740bb070 con 0x7f0d780a4580 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d6e7fc640 1 --2- 172.21.15.153:0/407237232 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d5c03cea0 0x7f0d5c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d6e7fc640 1 -- 172.21.15.153:0/407237232 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f0d740e87b0 con 0x7f0d780a4580 2024-09-10T20:26:22.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d6e7fc640 10 client.4227.objecter ms_dispatch 0x7f0d78000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d6e7fc640 3 client.4227.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d6e7fc640 3 client.4227.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d6e7fc640 20 client.4227.objecter dump_active .. 0 homeless 2024-09-10T20:26:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d7dd83640 1 --2- 172.21.15.153:0/407237232 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d5c03cea0 0x7f0d5c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d7dd83640 10 monclient: get_auth_request con 0x7f0d5c03cea0 auth_method 0 2024-09-10T20:26:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d6e7fc640 10 client.4227.objecter ms_handle_connect 0x7f0d5c03cea0 2024-09-10T20:26:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.562+0000 7f0d7dd83640 1 --2- 172.21.15.153:0/407237232 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d5c03cea0 0x7f0d5c03f360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d68004e30 tx=0x7f0d68009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.566+0000 7f0d6e7fc640 1 -- 172.21.15.153:0/407237232 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f0d74016130 con 0x7f0d780a4580 2024-09-10T20:26:22.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.566+0000 7f0d6e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.566+0000 7f0d6e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:22.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:22.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:22.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0d78122a10 con 0x7f0d780a4580 2024-09-10T20:26:22.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d6e7fc640 1 -- 172.21.15.153:0/407237232 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10987 (secure 0 0 0) 0x7f0d740bb360 con 0x7f0d780a4580 2024-09-10T20:26:22.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d6e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:22.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d6e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:22.740 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:22.740 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":20,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","created":"2024-09-10T20:25:59.587062+0000","modified":"2024-09-10T20:26:21.366705+0000","last_up_change":"2024-09-10T20:26:12.282988+0000","last_in_change":"2024-09-10T20:26:02.414479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:26:13.125072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-10T20:26:18.322750+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"faf817a9-b230-4970-b8c5-e5637fd29193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4216589932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":4216589932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":4216589932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4216589932}]},"public_addr":"172.21.15.153:6800/4216589932","cluster_addr":"172.21.15.153:6801/4216589932","heartbeat_back_addr":"172.21.15.153:6803/4216589932","heartbeat_front_addr":"172.21.15.153:6802/4216589932","state":["exists","up"]},{"osd":1,"uuid":"f85962f8-8bd9-4e26-8d66-49b6528617a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1715391555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":1715391555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1715391555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1715391555}]},"public_addr":"172.21.15.153:6804/1715391555","cluster_addr":"172.21.15.153:6805/1715391555","heartbeat_back_addr":"172.21.15.153:6807/1715391555","heartbeat_front_addr":"172.21.15.153:6806/1715391555","state":["exists","up"]},{"osd":2,"uuid":"18d54b55-1393-42b9-9fcc-84202659cf31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1626005750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":1626005750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":1626005750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1626005750}]},"public_addr":"172.21.15.153:6808/1626005750","cluster_addr":"172.21.15.153:6809/1626005750","heartbeat_back_addr":"172.21.15.153:6811/1626005750","heartbeat_front_addr":"172.21.15.153:6810/1626005750","state":["exists","up"]},{"osd":3,"uuid":"5149b9c4-35d7-4ffb-9fac-7b3d9b660101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2953083416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":2953083416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6811","nonce":2953083416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2953083416}]},"public_addr":"172.21.15.176:6808/2953083416","cluster_addr":"172.21.15.176:6809/2953083416","heartbeat_back_addr":"172.21.15.176:6811/2953083416","heartbeat_front_addr":"172.21.15.176:6810/2953083416","state":["exists","up"]},{"osd":4,"uuid":"61e0c0cc-4301-4947-9f1d-838d2d7bbfea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4030285519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6805","nonce":4030285519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6807","nonce":4030285519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4030285519}]},"public_addr":"172.21.15.176:6804/4030285519","cluster_addr":"172.21.15.176:6805/4030285519","heartbeat_back_addr":"172.21.15.176:6807/4030285519","heartbeat_front_addr":"172.21.15.176:6806/4030285519","state":["exists","up"]},{"osd":5,"uuid":"2feacd82-c45a-4d90-8210-ff9faecc2d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":2372911183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6801","nonce":2372911183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6803","nonce":2372911183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2372911183}]},"public_addr":"172.21.15.176:6800/2372911183","cluster_addr":"172.21.15.176:6801/2372911183","heartbeat_back_addr":"172.21.15.176:6803/2372911183","heartbeat_front_addr":"172.21.15.176:6802/2372911183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.301670+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:10.037323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.733818+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364748+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.173071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:26:22.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 10 librados: watch_flush enter 2024-09-10T20:26:22.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 10 librados: watch_flush exit 2024-09-10T20:26:22.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 20 client.4227.objecter shutdown clearing up homeless session... 2024-09-10T20:26:22.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 10 client.4227.objecter successfully canceled tick 2024-09-10T20:26:22.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d5c03cea0 msgr2=0x7f0d5c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 --2- 172.21.15.153:0/407237232 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d5c03cea0 0x7f0d5c03f360 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0d68004e30 tx=0x7f0d68009040 comp rx=0 tx=0).stop 2024-09-10T20:26:22.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 10 monclient: shutdown 2024-09-10T20:26:22.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:22.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d780a4580 msgr2=0x7f0d7809edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 --2- 172.21.15.153:0/407237232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d780a4580 0x7f0d7809edf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0d740ab000 tx=0x7f0d740191c0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 shutdown_connections 2024-09-10T20:26:22.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 --2- 172.21.15.153:0/407237232 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d5c03cea0 0x7f0d5c03f360 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 --2- 172.21.15.153:0/407237232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d780a49a0 0x7f0d78138860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 --2- 172.21.15.153:0/407237232 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d780a4580 0x7f0d7809edf0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.734+0000 7f0d7ffe7640 1 --2- 172.21.15.153:0/407237232 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d780a41a0 0x7f0d781219f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.738+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 >> 172.21.15.153:0/407237232 conn(0x7f0d780ab7e0 msgr2=0x7f0d780a10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:22.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.738+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 shutdown_connections 2024-09-10T20:26:22.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.738+0000 7f0d7ffe7640 1 -- 172.21.15.153:0/407237232 wait complete. 2024-09-10T20:26:22.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.738+0000 7f0d7ffe7640 1 librados: shutdown 2024-09-10T20:26:22.758 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-10T20:26:22.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-10T20:26:22.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-10T20:26:22.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-10T20:26:22.760 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-10T20:26:22.761 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-10T20:26:22.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient: get_monmap_and_config 2024-09-10T20:26:22.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient: monmap: 2024-09-10T20:26:22.877 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.877 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.877 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:22.876313+0000 2024-09-10T20:26:22.877 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:22.876313+0000 2024-09-10T20:26:22.877 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.877 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.878 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:22.878 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:22.878 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:22.878 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:22.878 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:22.878 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 1 Processor -- start 2024-09-10T20:26:22.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 1 -- start start 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient: init 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient: _add_conns ranks=[0,3,4,1,2] 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc3e80a41a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc3e80a4580 0x7fc3e813d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e80a4580 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fc3e80a49d0 0x7fc3e813fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e80a49d0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3e80a6a20 con 0x7fc3e80a41a0 2024-09-10T20:26:22.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fc3e80a2570 con 0x7fc3e80a4580 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fc3e8070400 con 0x7fc3e80a49d0 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ef6c8640 10 monclient(hunting): authenticate will time out at 943.015781s 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ecc63640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc3e80a4580 0x7fc3e813d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ed464640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fc3e80a41a0 0x7fc3e80a3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60436/0 (socket says 172.21.15.153:60436) 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ecc63640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc3e80a4580 0x7fc3e813d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42210/0 (socket says 172.21.15.153:42210) 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.870+0000 7fc3ed464640 1 -- 172.21.15.153:0/83271290 learned_addr learned my addr 172.21.15.153:0/83271290 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:22.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient(hunting): get_auth_request con 0x7fc3e80a4580 auth_method 0 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3edc65640 1 --2- 172.21.15.153:0/83271290 >> v2:172.21.15.176:3300/0 conn(0x7fc3e80a49d0 0x7fc3e813fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3edc65640 10 monclient(hunting): get_auth_request con 0x7fc3e80a49d0 auth_method 0 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3edc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3edc65640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3edc65640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient: get_monmap_and_config 2024-09-10T20:26:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient: monmap: 2024-09-10T20:26:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.884 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:22.878217+0000 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:22.878217+0000 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:22.885 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3edc65640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3edc65640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3edc65640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient(hunting): handle_auth_done global_id 4230 payload 274 2024-09-10T20:26:22.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 1 -- 172.21.15.153:0/83271290 >> v2:172.21.15.176:3300/0 conn(0x7fc3e80a49d0 msgr2=0x7fc3e813fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 1 --2- 172.21.15.153:0/83271290 >> v2:172.21.15.176:3300/0 conn(0x7fc3e80a49d0 0x7fc3e813fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 1 -- 172.21.15.153:0/83271290 >> v1:172.21.15.153:6789/0 conn(0x7fc3e80a41a0 legacy=0x7fc3e80a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 1 monclient: found mon.noname-a 2024-09-10T20:26:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7fc3ecc63640 1 -- 172.21.15.153:0/83271290 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e80a85a0 con 0x7fc3e80a4580 2024-09-10T20:26:22.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 1 Processor -- start 2024-09-10T20:26:22.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 1 -- start start 2024-09-10T20:26:22.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient: init 2024-09-10T20:26:22.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-10T20:26:22.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f340a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:22.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7f340a4580 0x7f7f3413d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f340a4580 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:22.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a49d0 0x7f7f3413fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient(hunting): picked mon.noname-b con 0x7f7f340a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a49d0 0x7f7f3413fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a49d0 0x7f7f3413fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55522/0 (socket says 172.21.15.153:55522) 2024-09-10T20:26:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 1 -- 172.21.15.153:0/2622733057 learned_addr learned my addr 172.21.15.153:0/2622733057 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f327fc640 1 --2- 172.21.15.153:0/2622733057 >> v2:172.21.15.176:3300/0 conn(0x7f7f340a4580 0x7f7f3413d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 1 -- 172.21.15.153:0/2622733057 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f340a6a20 con 0x7f7f340a41a0 2024-09-10T20:26:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 1 -- 172.21.15.153:0/2622733057 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7f340a2570 con 0x7f7f340a4580 2024-09-10T20:26:22.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 1 -- 172.21.15.153:0/2622733057 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7f34070400 con 0x7f7f340a49d0 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f3969d640 10 monclient(hunting): authenticate will time out at 943.018514s 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f327fc640 10 monclient(hunting): get_auth_request con 0x7f7f340a4580 auth_method 0 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f327fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient(hunting): get_auth_request con 0x7f7f340a49d0 auth_method 0 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.892 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient(hunting): handle_auth_done global_id 4214 payload 274 2024-09-10T20:26:22.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 1 -- 172.21.15.153:0/2622733057 >> v2:172.21.15.176:3300/0 conn(0x7f7f340a4580 msgr2=0x7f7f3413d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 1 --2- 172.21.15.153:0/2622733057 >> v2:172.21.15.176:3300/0 conn(0x7f7f340a4580 0x7f7f3413d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 1 -- 172.21.15.153:0/2622733057 >> v1:172.21.15.176:6789/0 conn(0x7f7f340a41a0 legacy=0x7f7f340a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 1 monclient: found mon.noname-b 2024-09-10T20:26:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.874+0000 7f7f337fe640 1 -- 172.21.15.153:0/2622733057 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f340a85a0 con 0x7f7f340a49d0 2024-09-10T20:26:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ecc63640 1 -- 172.21.15.153:0/83271290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a4580 msgr2=0x7fc3e813d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ecc63640 1 --2- 172.21.15.153:0/83271290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a4580 0x7fc3e813d7c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8070040 tx=0x7fc3d801c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5711ba7d3de0cc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 1 -- 172.21.15.153:0/83271290 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d80b0070 con 0x7fc3e80a4580 2024-09-10T20:26:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:22.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 10 monclient: dump: 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 1 -- 172.21.15.153:0/83271290 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3d80a9710 con 0x7fc3e80a4580 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 1 -- 172.21.15.153:0/83271290 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3d80a9ae0 con 0x7fc3e80a4580 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 5 monclient: authenticate success, global_id 4230 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 4 set_mon_vals no callback set 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3dffff640 10 monclient: dump: 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.897 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 10 monclient: shutdown 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/83271290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a4580 msgr2=0x7fc3e813d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 1 --2- 172.21.15.153:0/83271290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a4580 0x7fc3e813d7c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8070040 tx=0x7fc3d801c340 comp rx=0 tx=0).stop 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/83271290 shutdown_connections 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 1 --2- 172.21.15.153:0/83271290 >> v2:172.21.15.176:3300/0 conn(0x7fc3e80a49d0 0x7fc3e813fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 1 --2- 172.21.15.153:0/83271290 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a4580 0x7fc3e813d7c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/83271290 >> 172.21.15.153:0/83271290 conn(0x7fc3e80ab7e0 msgr2=0x7fc3e80a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:22.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/83271290 shutdown_connections 2024-09-10T20:26:22.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.878+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/83271290 wait complete. 2024-09-10T20:26:22.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f337fe640 1 -- 172.21.15.153:0/2622733057 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a49d0 msgr2=0x7f7f3413fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f337fe640 1 --2- 172.21.15.153:0/2622733057 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a49d0 0x7f7f3413fd10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009e50 tx=0x7f7f2c01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b55515daf4ca297f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 1 -- 172.21.15.153:0/2622733057 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0a5730 con 0x7f7f340a49d0 2024-09-10T20:26:22.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:22.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient: monmap: 2024-09-10T20:26:22.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.900 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:22.900 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:22.900 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:22.901 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.901 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.901 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:22.901 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:22.901 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:22.901 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 10 monclient: dump: 2024-09-10T20:26:22.901 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 1 -- 172.21.15.153:0/2622733057 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f2c0a58d0 con 0x7f7f340a49d0 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 1 -- 172.21.15.153:0/2622733057 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f2c0a5ca0 con 0x7f7f340a49d0 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f31ffb640 10 monclient: dump: 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.903 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.904 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.904 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 5 monclient: authenticate success, global_id 4214 2024-09-10T20:26:22.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:22.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 4 set_mon_vals no callback set 2024-09-10T20:26:22.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 librados: starting msgr at 2024-09-10T20:26:22.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 librados: starting objecter 2024-09-10T20:26:22.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 Processor -- start 2024-09-10T20:26:22.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 -- start start 2024-09-10T20:26:22.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 librados: setting wanted keys 2024-09-10T20:26:22.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 librados: calling monclient init 2024-09-10T20:26:22.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient: init 2024-09-10T20:26:22.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient: shutdown 2024-09-10T20:26:22.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:22.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 -- 172.21.15.153:0/2622733057 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a49d0 msgr2=0x7f7f3413fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 --2- 172.21.15.153:0/2622733057 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a49d0 0x7f7f3413fd10 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7f2c009e50 tx=0x7f7f2c01c8f0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 -- 172.21.15.153:0/2622733057 shutdown_connections 2024-09-10T20:26:22.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 --2- 172.21.15.153:0/2622733057 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a49d0 0x7f7f3413fd10 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 --2- 172.21.15.153:0/2622733057 >> v2:172.21.15.176:3300/0 conn(0x7f7f340a4580 0x7f7f3413d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 -- 172.21.15.153:0/2622733057 >> 172.21.15.153:0/2622733057 conn(0x7f7f340ab7e0 msgr2=0x7f7f340a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:22.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:26:22.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a41a0 0x7fc3e8128980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc3e80a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:22.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3e80a4580 0x7fc3e81cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc3e80a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:22.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3e80a49d0 0x7fc3e813f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc3e80a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:22.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e8142e00 con 0x7fc3e80a41a0 2024-09-10T20:26:22.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc3e8142c80 con 0x7fc3e80a4580 2024-09-10T20:26:22.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc3e8142f80 con 0x7fc3e80a49d0 2024-09-10T20:26:22.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ef6c8640 10 monclient(hunting): authenticate will time out at 943.025890s 2024-09-10T20:26:22.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 -- 172.21.15.153:0/2622733057 shutdown_connections 2024-09-10T20:26:22.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3edc65640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3e80a49d0 0x7fc3e813f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a41a0 0x7fc3e8128980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3edc65640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3e80a49d0 0x7fc3e813f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55536/0 (socket says 172.21.15.153:55536) 2024-09-10T20:26:22.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3edc65640 1 -- 172.21.15.153:0/1384079199 learned_addr learned my addr 172.21.15.153:0/1384079199 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:22.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3edc65640 10 monclient(hunting): get_auth_request con 0x7fc3e80a49d0 auth_method 0 2024-09-10T20:26:22.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3edc65640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3edc65640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3edc65640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 -- 172.21.15.153:0/2622733057 wait complete. 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient(hunting): get_auth_request con 0x7fc3e80a41a0 auth_method 0 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ecc63640 1 --2- 172.21.15.153:0/1384079199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3e80a4580 0x7fc3e81cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ecc63640 10 monclient(hunting): get_auth_request con 0x7fc3e80a4580 auth_method 0 2024-09-10T20:26:22.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ecc63640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ecc63640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ecc63640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ecc63640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ecc63640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ecc63640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 1 -- 172.21.15.153:0/1384079199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3e80a49d0 msgr2=0x7fc3e813f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 1 --2- 172.21.15.153:0/1384079199 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc3e80a49d0 0x7fc3e813f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 1 -- 172.21.15.153:0/1384079199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3e80a4580 msgr2=0x7fc3e81cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 1 --2- 172.21.15.153:0/1384079199 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc3e80a4580 0x7fc3e81cd5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 1 monclient: found mon.noname-a 2024-09-10T20:26:22.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7fc3ed464640 1 -- 172.21.15.153:0/1384079199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3e8127e20 con 0x7fc3e80a41a0 2024-09-10T20:26:22.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient: monmap: 2024-09-10T20:26:22.915 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.915 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:22.915 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:22.915 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:22.915 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 librados: starting msgr at 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 librados: starting objecter 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 Processor -- start 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 -- start start 2024-09-10T20:26:22.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 librados: setting wanted keys 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 librados: calling monclient init 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient: init 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f340a41a0 0x7f7f34128460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient(hunting): picked mon.noname-a con 0x7f7f340a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a4580 0x7f7f34126300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f340a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f340a49d0 0x7f7f3413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient(hunting): picked mon.noname-b con 0x7f7f340a49d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f34142e00 con 0x7f7f340a41a0 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7f34142c80 con 0x7f7f340a49d0 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7f34142f80 con 0x7f7f340a4580 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f3969d640 10 monclient(hunting): authenticate will time out at 943.027836s 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f327fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a4580 0x7f7f34126300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.882+0000 7f7f32ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f340a41a0 0x7f7f34128460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f340a41a0 0x7f7f34128460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42226/0 (socket says 172.21.15.153:42226) 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a4580 0x7f7f34126300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55542/0 (socket says 172.21.15.153:55542) 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 1 -- 172.21.15.153:0/2217099732 learned_addr learned my addr 172.21.15.153:0/2217099732 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:22.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 10 monclient(hunting): get_auth_request con 0x7f7f340a41a0 auth_method 0 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient(hunting): get_auth_request con 0x7f7f340a4580 auth_method 0 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f337fe640 1 --2- 172.21.15.153:0/2217099732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f340a49d0 0x7f7f3413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f337fe640 10 monclient(hunting): get_auth_request con 0x7f7f340a49d0 auth_method 0 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f337fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f337fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f337fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f337fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f337fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f337fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 1 -- 172.21.15.153:0/2217099732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f340a49d0 msgr2=0x7f7f3413f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 1 --2- 172.21.15.153:0/2217099732 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f340a49d0 0x7f7f3413f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 1 -- 172.21.15.153:0/2217099732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f340a41a0 msgr2=0x7f7f34128460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 1 --2- 172.21.15.153:0/2217099732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f340a41a0 0x7f7f34128460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 1 monclient: found mon.noname-c 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 1 -- 172.21.15.153:0/2217099732 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f340a21c0 con 0x7f7f340a4580 2024-09-10T20:26:22.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f32ffd640 1 --2- 172.21.15.153:0/2217099732 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f340a41a0 0x7f7f34128460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ed464640 1 -- 172.21.15.153:0/1384079199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a41a0 msgr2=0x7fc3e8128980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 client.?.objecter ms_handle_connect 0x7fc3e80a41a0 2024-09-10T20:26:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ed464640 1 --2- 172.21.15.153:0/1384079199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a41a0 0x7fc3e8128980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c410 tx=0x7fc3e400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 1 -- 172.21.15.153:0/1384079199 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4017070 con 0x7fc3e80a41a0 2024-09-10T20:26:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: dump: 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 5 monclient: authenticate success, global_id 4233 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 10 monclient: _renew_subs 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3e81298e0 con 0x7fc3e80a41a0 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 10 monclient: _renew_subs 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3e81cdb10 con 0x7fc3e80a41a0 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 1 librados: init done 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 1 -- 172.21.15.153:0/1384079199 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3e40070d0 con 0x7fc3e80a41a0 2024-09-10T20:26:22.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 1 -- 172.21.15.153:0/1384079199 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3e40074c0 con 0x7fc3e80a41a0 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: dump: 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.926 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.927 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3df7fe640 4 set_mon_vals no callback set 2024-09-10T20:26:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 10 client.4233.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 15 client.4233.objecter _session_command_op_assign -1 1 2024-09-10T20:26:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 20 client.4233.objecter _get_session osd=-1 returning homeless 2024-09-10T20:26:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 10 client.4233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:22.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3c8002920 con 0x7fc3e80a41a0 2024-09-10T20:26:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 1 -- 172.21.15.153:0/2217099732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a4580 msgr2=0x7f7f34126300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f327fc640 1 --2- 172.21.15.153:0/2217099732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a4580 0x7f7f34126300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800e870 tx=0x7f7f2800ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 client.?.objecter ms_handle_connect 0x7f7f340a4580 2024-09-10T20:26:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:22.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 1 -- 172.21.15.153:0/2217099732 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28017070 con 0x7f7f340a4580 2024-09-10T20:26:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 monclient: dump: 2024-09-10T20:26:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.929 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 1 -- 172.21.15.153:0/2217099732 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f2800cb50 con 0x7f7f340a4580 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 1 -- 172.21.15.153:0/2217099732 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f28013470 con 0x7f7f340a4580 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f1b7fe640 10 monclient: dump: 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.931 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f3969d640 5 monclient: authenticate success, global_id 4220 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f317fa640 4 set_mon_vals no callback set 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f3969d640 10 monclient: _renew_subs 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f3969d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f34127900 con 0x7f7f340a4580 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f3969d640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f3969d640 10 monclient: _renew_subs 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f3969d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f3413ff70 con 0x7f7f340a4580 2024-09-10T20:26:22.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7f7f3969d640 1 librados: init done 2024-09-10T20:26:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 1 -- 172.21.15.153:0/1384079199 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fc3e401a070 con 0x7fc3e80a41a0 2024-09-10T20:26:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 1 --2- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc3cc03ced0 0x7fc3cc03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 1 -- 172.21.15.153:0/1384079199 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7fc3e404e6b0 con 0x7fc3e80a41a0 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 client.4233.objecter ms_dispatch 0x7fc3e8000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 3 client.4233.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 3 client.4233.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 client.4233.objecter checking command 1 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 1 --2- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6804/4030285519 conn(0x7fc3cc042970 0x7fc3cc044db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 20 client.4233.objecter _get_session s=0x7fc3cc0422c0 osd=4 3 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 20 client.4233.objecter put_session s=0x7fc3cc0422c0 osd=4 3 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 15 client.4233.objecter _session_command_op_remove -1 1 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 20 client.4233.objecter _get_session s=0x7fc3cc0422c0 osd=4 3 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 20 client.4233.objecter get_session s=0x7fc3cc0422c0 osd=4 3 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 15 client.4233.objecter _session_command_op_assign 4 1 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 20 client.4233.objecter put_session s=0x7fc3cc0422c0 osd=4 4 2024-09-10T20:26:22.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 client.4233.objecter _send_command 1 2024-09-10T20:26:22.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 1 -- 172.21.15.153:0/1384079199 --> v2:172.21.15.176:6804/4030285519 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3cc04dc50 con 0x7fc3cc042970 2024-09-10T20:26:22.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 20 client.4233.objecter dump_active .. 0 homeless 2024-09-10T20:26:22.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 1 -- 172.21.15.153:0/1384079199 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc3e4055050 con 0x7fc3e80a41a0 2024-09-10T20:26:22.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.886+0000 7fc3ddffb640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-10T20:26:22.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7fc3ecc63640 1 --2- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc3cc03ced0 0x7fc3cc03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7fc3ecc63640 10 monclient: get_auth_request con 0x7fc3cc03ced0 auth_method 0 2024-09-10T20:26:22.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7fc3edc65640 1 --2- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6804/4030285519 conn(0x7fc3cc042970 0x7fc3cc044db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7fc3edc65640 10 monclient: get_auth_request con 0x7fc3cc042970 auth_method 0 2024-09-10T20:26:22.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f3969d640 10 client.4220.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f3969d640 15 client.4220.objecter _session_command_op_assign -1 1 2024-09-10T20:26:22.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f3969d640 20 client.4220.objecter _get_session osd=-1 returning homeless 2024-09-10T20:26:22.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f3969d640 10 client.4220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f3969d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7f34129480 con 0x7f7f340a4580 2024-09-10T20:26:22.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f327fc640 1 --2- 172.21.15.153:0/2217099732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a4580 0x7f7f34126300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800e870 tx=0x7f7f2800ed30 comp rx=0 tx=0).queue_received will delay after 1.726e+09 on 0x7f7f2804f260 osd_map(20..20 src has 1..20) 2024-09-10T20:26:22.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 1 -- 172.21.15.153:0/2217099732 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f7f2801a070 con 0x7f7f340a4580 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 1 --2- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7f2003cef0 0x7f7f2003f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 1 -- 172.21.15.153:0/2217099732 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f7f2804f260 con 0x7f7f340a4580 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 10 client.4220.objecter ms_dispatch 0x7f7f34000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 3 client.4220.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 3 client.4220.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 10 client.4220.objecter checking command 1 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 1 --2- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6808/2953083416 conn(0x7f7f20042a00 0x7f7f20044e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 20 client.4220.objecter _get_session s=0x7f7f20042350 osd=3 3 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 20 client.4220.objecter put_session s=0x7f7f20042350 osd=3 3 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 15 client.4220.objecter _session_command_op_remove -1 1 2024-09-10T20:26:22.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 20 client.4220.objecter _get_session s=0x7f7f20042350 osd=3 3 2024-09-10T20:26:22.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 20 client.4220.objecter get_session s=0x7f7f20042350 osd=3 3 2024-09-10T20:26:22.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 15 client.4220.objecter _session_command_op_assign 3 1 2024-09-10T20:26:22.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 20 client.4220.objecter put_session s=0x7f7f20042350 osd=3 4 2024-09-10T20:26:22.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 10 client.4220.objecter _send_command 1 2024-09-10T20:26:22.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 1 -- 172.21.15.153:0/2217099732 --> v2:172.21.15.176:6808/2953083416 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f200454d0 con 0x7f7f20042a00 2024-09-10T20:26:22.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f1b7fe640 20 client.4220.objecter dump_active .. 0 homeless 2024-09-10T20:26:22.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f32ffd640 1 --2- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7f2003cef0 0x7f7f2003f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f32ffd640 10 monclient: get_auth_request con 0x7f7f2003cef0 auth_method 0 2024-09-10T20:26:22.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f337fe640 1 --2- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6808/2953083416 conn(0x7f7f20042a00 0x7f7f20044e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7f7f337fe640 10 monclient: get_auth_request con 0x7f7f20042a00 auth_method 0 2024-09-10T20:26:22.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7fc3ecc63640 1 --2- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc3cc03ced0 0x7fc3cc03f390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80ab000 tx=0x7fc3d80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.890+0000 7fc3ddffb640 10 client.4233.objecter ms_handle_connect 0x7fc3cc03ced0 2024-09-10T20:26:22.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7f7f1b7fe640 10 client.4220.objecter ms_handle_connect 0x7f7f2003cef0 2024-09-10T20:26:22.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7f7f32ffd640 1 --2- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7f2003cef0 0x7f7f2003f3b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7f24004470 tx=0x7f7f24003700 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7fc3edc65640 1 --2- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6804/4030285519 conn(0x7fc3cc042970 0x7fc3cc044db0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7fc3ddffb640 10 client.4233.objecter ms_handle_connect 0x7fc3cc042970 2024-09-10T20:26:22.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7f7f337fe640 1 --2- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6808/2953083416 conn(0x7f7f20042a00 0x7f7f20044e40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7f7f1b7fe640 10 client.4220.objecter ms_handle_connect 0x7f7f20042a00 2024-09-10T20:26:22.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7fc3ddffb640 1 -- 172.21.15.153:0/1384079199 <== osd.4 v2:172.21.15.176:6804/4030285519 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3cc04dc50 con 0x7fc3cc042970 2024-09-10T20:26:22.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7fc3ddffb640 10 client.4233.objecter ms_dispatch 0x7fc3e8000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:26:22.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7fc3ddffb640 10 client.4233.objecter _finish_command 1 = system:0 2024-09-10T20:26:22.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7fc3ddffb640 20 client.4233.objecter put_session s=0x7fc3cc0422c0 osd=4 4 2024-09-10T20:26:22.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.894+0000 7fc3ddffb640 15 client.4233.objecter _session_command_op_remove 4 1 2024-09-10T20:26:22.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.898+0000 7f7f1b7fe640 1 -- 172.21.15.153:0/2217099732 <== osd.3 v2:172.21.15.176:6808/2953083416 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7f200454d0 con 0x7f7f20042a00 2024-09-10T20:26:22.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.898+0000 7f7f1b7fe640 10 client.4220.objecter ms_dispatch 0x7f7f34000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:26:22.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.898+0000 7f7f1b7fe640 10 client.4220.objecter _finish_command 1 = system:0 2024-09-10T20:26:22.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.898+0000 7f7f1b7fe640 20 client.4220.objecter put_session s=0x7f7f20042350 osd=3 4 2024-09-10T20:26:22.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.898+0000 7f7f1b7fe640 15 client.4220.objecter _session_command_op_remove 3 1 2024-09-10T20:26:22.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient: get_monmap_and_config 2024-09-10T20:26:22.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient: monmap: 2024-09-10T20:26:22.944 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:22.913991+0000 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:22.913991+0000 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:22.945 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient: get_monmap_and_config 2024-09-10T20:26:22.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient: monmap: 2024-09-10T20:26:22.946 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.946 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:22.914291+0000 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:22.914291+0000 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:22.947 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 1 Processor -- start 2024-09-10T20:26:22.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 1 -- start start 2024-09-10T20:26:22.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient: init 2024-09-10T20:26:22.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 1 Processor -- start 2024-09-10T20:26:22.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-10T20:26:22.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7efc800a41a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:22.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7efc800a4580 0x7efc8013d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient(hunting): picked mon.noname-c con 0x7efc800a4580 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:22.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efc800a49d0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:22.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 1 -- start start 2024-09-10T20:26:22.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient: init 2024-09-10T20:26:22.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-10T20:26:22.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f408c0a1380 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:22.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c0a1760 0x7f408c0a2770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient(hunting): picked mon.noname-b con 0x7f408c0a1760 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:22.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f408c0a0180 0x7f408c144670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient(hunting): picked mon.noname-c con 0x7f408c0a0180 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:22.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f408c0a32f0 con 0x7f408c0a1380 2024-09-10T20:26:22.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f408c0a3730 con 0x7f408c0a0180 2024-09-10T20:26:22.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f408c09f4c0 con 0x7f408c0a1760 2024-09-10T20:26:22.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4093f9e640 10 monclient(hunting): authenticate will time out at 943.053680s 2024-09-10T20:26:22.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc85f8b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7efc800a41a0 0x7efc800a3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60448/0 (socket says 172.21.15.153:60448) 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc85f8b640 1 -- 172.21.15.153:0/1433532023 learned_addr learned my addr 172.21.15.153:0/1433532023 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 1 -- 172.21.15.153:0/1433532023 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc800a6a20 con 0x7efc800a41a0 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 1 -- 172.21.15.153:0/1433532023 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc800a85a0 con 0x7efc800a49d0 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 1 -- 172.21.15.153:0/1433532023 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7efc800a2570 con 0x7efc800a4580 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc881ef640 10 monclient(hunting): authenticate will time out at 943.053931s 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 1 --2- 172.21.15.153:0/1433532023 >> v2:172.21.15.176:3300/0 conn(0x7efc800a4580 0x7efc8013d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient(hunting): get_auth_request con 0x7efc800a4580 auth_method 0 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c0a1760 0x7f408c0a2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c0a1760 0x7f408c0a2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55554/0 (socket says 172.21.15.153:55554) 2024-09-10T20:26:22.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 1 -- 172.21.15.153:0/2303970751 learned_addr learned my addr 172.21.15.153:0/2303970751 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:22.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient(hunting): get_auth_request con 0x7f408c0a1760 auth_method 0 2024-09-10T20:26:22.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2024-09-10T20:26:22.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 1 -- 172.21.15.153:0/2303970751 >> v2:172.21.15.176:3300/0 conn(0x7f408c0a0180 msgr2=0x7f408c144670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 1 --2- 172.21.15.153:0/2303970751 >> v2:172.21.15.176:3300/0 conn(0x7f408c0a0180 0x7f408c144670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 1 -- 172.21.15.153:0/2303970751 >> v1:172.21.15.176:6789/0 conn(0x7f408c0a1380 legacy=0x7f408c0a2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 1 monclient: found mon.noname-b 2024-09-10T20:26:22.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7f4091539640 1 -- 172.21.15.153:0/2303970751 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c09f7a0 con 0x7f408c0a1760 2024-09-10T20:26:22.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient(hunting): handle_auth_done global_id 4219 payload 274 2024-09-10T20:26:22.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 1 -- 172.21.15.153:0/1433532023 >> v1:172.21.15.176:6789/0 conn(0x7efc800a49d0 legacy=0x7efc8013fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 1 -- 172.21.15.153:0/1433532023 >> v1:172.21.15.153:6789/0 conn(0x7efc800a41a0 legacy=0x7efc800a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:22.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 1 monclient: found mon.noname-c 2024-09-10T20:26:22.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:26:22.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.910+0000 7efc8578a640 1 -- 172.21.15.153:0/1433532023 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc80143450 con 0x7efc800a4580 2024-09-10T20:26:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4091539640 1 -- 172.21.15.153:0/2303970751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c0a1760 msgr2=0x7f408c0a2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4091539640 1 --2- 172.21.15.153:0/2303970751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c0a1760 0x7f408c0a2770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f408c0a36d0 tx=0x7f407c01c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6783d74a63490d76 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 1 -- 172.21.15.153:0/2303970751 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c0b1070 con 0x7f408c0a1760 2024-09-10T20:26:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 10 monclient: dump: 2024-09-10T20:26:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.958 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 1 -- 172.21.15.153:0/2303970751 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c0a2ec0 con 0x7f408c0a1760 2024-09-10T20:26:22.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 1 -- 172.21.15.153:0/2303970751 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f407c0ac820 con 0x7f408c0a1760 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4090d38640 10 monclient: dump: 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.960 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.961 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.961 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.961 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc8578a640 1 -- 172.21.15.153:0/1433532023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc800a4580 msgr2=0x7efc8013d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc8578a640 1 --2- 172.21.15.153:0/1433532023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc800a4580 0x7efc8013d7c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efc700097c0 tx=0x7efc7001c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0f93baf3198904e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 1 -- 172.21.15.153:0/1433532023 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc700b0070 con 0x7efc800a4580 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 10 monclient: dump: 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc881ef640 5 monclient: authenticate success, global_id 4219 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc881ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 1 -- 172.21.15.153:0/1433532023 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc700a97f0 con 0x7efc800a4580 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc881ef640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 1 -- 172.21.15.153:0/1433532023 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc700a9bc0 con 0x7efc800a4580 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc881ef640 4 set_mon_vals no callback set 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:22.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7efc84f89640 10 monclient: dump: 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.964 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4093f9e640 5 monclient: authenticate success, global_id 4223 2024-09-10T20:26:22.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4093f9e640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:22.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.914+0000 7f4093f9e640 4 set_mon_vals no callback set 2024-09-10T20:26:22.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient: shutdown 2024-09-10T20:26:22.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:22.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 -- 172.21.15.153:0/2303970751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c0a1760 msgr2=0x7f408c0a2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 --2- 172.21.15.153:0/2303970751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c0a1760 0x7f408c0a2770 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f408c0a36d0 tx=0x7f407c01c400 comp rx=0 tx=0).stop 2024-09-10T20:26:22.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 -- 172.21.15.153:0/2303970751 shutdown_connections 2024-09-10T20:26:22.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 --2- 172.21.15.153:0/2303970751 >> v2:172.21.15.176:3300/0 conn(0x7f408c0a0180 0x7f408c144670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 --2- 172.21.15.153:0/2303970751 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c0a1760 0x7f408c0a2770 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 -- 172.21.15.153:0/2303970751 >> 172.21.15.153:0/2303970751 conn(0x7f408c0a7830 msgr2=0x7f408c0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:22.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 -- 172.21.15.153:0/2303970751 shutdown_connections 2024-09-10T20:26:22.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient: shutdown 2024-09-10T20:26:22.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:22.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 -- 172.21.15.153:0/1433532023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc800a4580 msgr2=0x7efc8013d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 --2- 172.21.15.153:0/1433532023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc800a4580 0x7efc8013d7c0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efc700097c0 tx=0x7efc7001c360 comp rx=0 tx=0).stop 2024-09-10T20:26:22.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 -- 172.21.15.153:0/1433532023 shutdown_connections 2024-09-10T20:26:22.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 --2- 172.21.15.153:0/1433532023 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc800a4580 0x7efc8013d7c0 secure :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efc700097c0 tx=0x7efc7001c360 comp rx=0 tx=0).stop 2024-09-10T20:26:22.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 -- 172.21.15.153:0/1433532023 >> 172.21.15.153:0/1433532023 conn(0x7efc800ab7e0 msgr2=0x7efc800a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:22.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 -- 172.21.15.153:0/2303970751 wait complete. 2024-09-10T20:26:22.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient: monmap: 2024-09-10T20:26:22.967 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.967 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:22.967 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 librados: starting msgr at 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 librados: starting objecter 2024-09-10T20:26:22.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 Processor -- start 2024-09-10T20:26:22.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 -- 172.21.15.153:0/1433532023 shutdown_connections 2024-09-10T20:26:22.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 -- start start 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 librados: setting wanted keys 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 librados: calling monclient init 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient: init 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0a0180 0x7f408c1dd900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient(hunting): picked mon.noname-a con 0x7f408c0a0180 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408c0a1380 0x7f408c1dde40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient(hunting): picked mon.noname-b con 0x7f408c0a1380 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:22.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c1de380 0x7f408c091df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient(hunting): picked mon.noname-c con 0x7f408c1de380 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:22.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f408c145800 con 0x7f408c0a0180 2024-09-10T20:26:22.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f408c145680 con 0x7f408c0a1380 2024-09-10T20:26:22.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f408c145980 con 0x7f408c1de380 2024-09-10T20:26:22.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4093f9e640 10 monclient(hunting): authenticate will time out at 943.062695s 2024-09-10T20:26:22.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 -- 172.21.15.153:0/1433532023 wait complete. 2024-09-10T20:26:22.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient: monmap: 2024-09-10T20:26:22.972 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.972 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:22.972 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:22.972 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:22.973 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.973 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.973 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:22.973 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:22.973 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:22.973 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 librados: starting msgr at 2024-09-10T20:26:22.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 librados: starting objecter 2024-09-10T20:26:22.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 Processor -- start 2024-09-10T20:26:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c1de380 0x7f408c091df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c1de380 0x7f408c091df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55562/0 (socket says 172.21.15.153:55562) 2024-09-10T20:26:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 1 -- 172.21.15.153:0/1915786036 learned_addr learned my addr 172.21.15.153:0/1915786036 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient(hunting): get_auth_request con 0x7f408c1de380 auth_method 0 2024-09-10T20:26:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091d3a640 1 --2- 172.21.15.153:0/1915786036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0a0180 0x7f408c1dd900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091d3a640 10 monclient(hunting): get_auth_request con 0x7f408c0a0180 auth_method 0 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091d3a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091d3a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091d3a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091539640 1 --2- 172.21.15.153:0/1915786036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408c0a1380 0x7f408c1dde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091539640 10 monclient(hunting): get_auth_request con 0x7f408c0a1380 auth_method 0 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091539640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091539640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091539640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091539640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091539640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f4091539640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 1 -- 172.21.15.153:0/1915786036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408c0a1380 msgr2=0x7f408c1dde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 1 --2- 172.21.15.153:0/1915786036 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f408c0a1380 0x7f408c1dde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 1 -- 172.21.15.153:0/1915786036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0a0180 msgr2=0x7f408c1dd900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 1 --2- 172.21.15.153:0/1915786036 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f408c0a0180 0x7f408c1dd900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 1 monclient: found mon.noname-c 2024-09-10T20:26:22.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7f409253b640 1 -- 172.21.15.153:0/1915786036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f408c09e650 con 0x7f408c1de380 2024-09-10T20:26:22.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 -- start start 2024-09-10T20:26:22.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 librados: setting wanted keys 2024-09-10T20:26:22.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 librados: calling monclient init 2024-09-10T20:26:22.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient: init 2024-09-10T20:26:22.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:26:22.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc800a41a0 0x7efc80132630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient(hunting): picked mon.noname-b con 0x7efc800a41a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc800a49d0 0x7efc80132b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient(hunting): picked mon.noname-c con 0x7efc800a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc801330b0 0x7efc801dd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient(hunting): picked mon.noname-a con 0x7efc801330b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7efc80144040 con 0x7efc801330b0 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7efc80070400 con 0x7efc800a41a0 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7efc801441c0 con 0x7efc800a49d0 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc881ef640 10 monclient(hunting): authenticate will time out at 943.063581s 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc8578a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc800a49d0 0x7efc80132b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc8578a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc800a49d0 0x7efc80132b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55566/0 (socket says 172.21.15.153:55566) 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc8578a640 1 -- 172.21.15.153:0/1726908079 learned_addr learned my addr 172.21.15.153:0/1726908079 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc8578a640 10 monclient(hunting): get_auth_request con 0x7efc800a49d0 auth_method 0 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc8578a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc8578a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc8578a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc85f8b640 1 --2- 172.21.15.153:0/1726908079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc800a41a0 0x7efc80132630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.918+0000 7efc8678c640 1 --2- 172.21.15.153:0/1726908079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc801330b0 0x7efc801dd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc85f8b640 10 monclient(hunting): get_auth_request con 0x7efc800a41a0 auth_method 0 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc85f8b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc85f8b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc85f8b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8678c640 10 monclient(hunting): get_auth_request con 0x7efc801330b0 auth_method 0 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8678c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:22.981 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8678c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8678c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc85f8b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc85f8b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc85f8b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8678c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8678c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8678c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 10 monclient(hunting): handle_auth_done global_id 4229 payload 824 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 1 -- 172.21.15.153:0/1726908079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc800a41a0 msgr2=0x7efc80132630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 1 --2- 172.21.15.153:0/1726908079 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7efc800a41a0 0x7efc80132630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 1 -- 172.21.15.153:0/1726908079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc801330b0 msgr2=0x7efc801dd2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:22.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 1 --2- 172.21.15.153:0/1726908079 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7efc801330b0 0x7efc801dd2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:22.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 10 monclient: _finish_hunting 0 2024-09-10T20:26:22.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 1 monclient: found mon.noname-c 2024-09-10T20:26:22.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:22.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 1 -- 172.21.15.153:0/1726908079 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc80135d30 con 0x7efc800a49d0 2024-09-10T20:26:22.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient: get_monmap_and_config 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient: build_initial_monmap 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient: monmap: 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:22.928384+0000 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:22.928384+0000 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:22.984 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 1 -- 172.21.15.153:0/1726908079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc800a49d0 msgr2=0x7efc80132b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc8578a640 1 --2- 172.21.15.153:0/1726908079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc800a49d0 0x7efc80132b70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efc700ae040 tx=0x7efc7001aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 client.?.objecter ms_handle_connect 0x7efc800a49d0 2024-09-10T20:26:22.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:22.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f409253b640 1 -- 172.21.15.153:0/1915786036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c1de380 msgr2=0x7f408c091df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:22.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f409253b640 1 --2- 172.21.15.153:0/1915786036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c1de380 0x7f408c091df0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f408400b9e0 tx=0x7f408400bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:22.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 1 Processor -- start 2024-09-10T20:26:22.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 client.?.objecter ms_handle_connect 0x7f408c1de380 2024-09-10T20:26:22.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:22.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 1 -- start start 2024-09-10T20:26:22.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient: init 2024-09-10T20:26:22.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 1 -- 172.21.15.153:0/1726908079 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc700b0050 con 0x7efc800a49d0 2024-09-10T20:26:22.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:22.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 monclient: dump: 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.988 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 1 -- 172.21.15.153:0/1726908079 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc700bc070 con 0x7efc800a49d0 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 1 -- 172.21.15.153:0/1726908079 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc700abe30 con 0x7efc800a49d0 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc767fc640 10 monclient: dump: 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.989 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc77fff640 4 set_mon_vals no callback set 2024-09-10T20:26:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 1 -- 172.21.15.153:0/1915786036 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084019070 con 0x7f408c1de380 2024-09-10T20:26:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 monclient: dump: 2024-09-10T20:26:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.990 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 monclient: _finish_auth 0 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 monclient: _check_auth_tickets 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 1 -- 172.21.15.153:0/1915786036 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408400cd50 con 0x7f408c1de380 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:22.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 1 -- 172.21.15.153:0/1915786036 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4084014680 con 0x7f408c1de380 2024-09-10T20:26:22.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:22.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc881ef640 5 monclient: authenticate success, global_id 4229 2024-09-10T20:26:22.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:22.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40827fc640 10 monclient: dump: 2024-09-10T20:26:22.992 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:22.992 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:22.992 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.992 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:22.992 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc881ef640 10 monclient: _renew_subs 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc881ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc80135a50 con 0x7efc800a49d0 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc881ef640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc881ef640 10 monclient: _renew_subs 2024-09-10T20:26:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc881ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7efc80133b20 con 0x7efc800a49d0 2024-09-10T20:26:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7efc881ef640 1 librados: init done 2024-09-10T20:26:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f4093f9e640 5 monclient: authenticate success, global_id 4226 2024-09-10T20:26:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f4093f9e640 10 monclient: _renew_subs 2024-09-10T20:26:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f4093f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f408c0938b0 con 0x7f408c1de380 2024-09-10T20:26:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f40837fe640 4 set_mon_vals no callback set 2024-09-10T20:26:22.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f4093f9e640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f4093f9e640 10 monclient: _renew_subs 2024-09-10T20:26:22.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f4093f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f408c095c10 con 0x7f408c1de380 2024-09-10T20:26:22.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f4093f9e640 1 librados: init done 2024-09-10T20:26:22.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:22.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-10T20:26:22.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6c580a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:22.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f6c580a4580 0x7f6c5813d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:22.995 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient(hunting): picked mon.noname-a con 0x7f6c580a4580 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6c580a49d0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c580a6a20 con 0x7f6c580a49d0 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c580a85a0 con 0x7f6c580a41a0 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f6c580a2570 con 0x7f6c580a4580 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.922+0000 7f6c60475640 10 monclient(hunting): authenticate will time out at 943.067992s 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc881ef640 10 client.4229.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc881ef640 15 client.4229.objecter _session_command_op_assign -1 1 2024-09-10T20:26:22.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc881ef640 20 client.4229.objecter _get_session osd=-1 returning homeless 2024-09-10T20:26:22.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc881ef640 10 client.4229.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc881ef640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efc800a0cf0 con 0x7efc800a49d0 2024-09-10T20:26:22.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 10 client.4233.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T20:26:22.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 15 client.4233.objecter _session_command_op_assign -1 2 2024-09-10T20:26:22.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 20 client.4233.objecter _get_session s=0x7fc3cc0422c0 osd=4 3 2024-09-10T20:26:22.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 20 client.4233.objecter put_session s=0x7fc3cc0422c0 osd=4 3 2024-09-10T20:26:22.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 20 client.4233.objecter _get_session s=0x7fc3cc0422c0 osd=4 3 2024-09-10T20:26:22.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 15 client.4233.objecter _session_command_op_remove -1 2 2024-09-10T20:26:22.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 20 client.4233.objecter get_session s=0x7fc3cc0422c0 osd=4 3 2024-09-10T20:26:22.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 15 client.4233.objecter _session_command_op_assign 4 2 2024-09-10T20:26:22.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 20 client.4233.objecter put_session s=0x7fc3cc0422c0 osd=4 4 2024-09-10T20:26:22.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 10 client.4233.objecter _send_command 2 2024-09-10T20:26:22.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 --> v2:172.21.15.176:6804/4030285519 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc3c8001230 con 0x7fc3cc042970 2024-09-10T20:26:22.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5ea12640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6c580a49d0 0x7f6c5813fd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60458/0 (socket says 172.21.15.153:60458) 2024-09-10T20:26:22.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5ea12640 1 -- 172.21.15.153:0/3375166969 learned_addr learned my addr 172.21.15.153:0/3375166969 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:22.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 1 --2- 172.21.15.153:0/3375166969 >> v2:172.21.15.153:3300/0 conn(0x7f6c580a4580 0x7f6c5813d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:22.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4093f9e640 10 client.4226.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:22.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4093f9e640 15 client.4226.objecter _session_command_op_assign -1 1 2024-09-10T20:26:22.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4093f9e640 20 client.4226.objecter _get_session osd=-1 returning homeless 2024-09-10T20:26:22.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4093f9e640 10 client.4226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:22.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4093f9e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:22.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f408c0a2ec0 con 0x7f408c1de380 2024-09-10T20:26:23.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient(hunting): get_auth_request con 0x7f6c580a4580 auth_method 0 2024-09-10T20:26:23.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ddffb640 1 -- 172.21.15.153:0/1384079199 <== osd.4 v2:172.21.15.176:6804/4030285519 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc3c8001230 con 0x7fc3cc042970 2024-09-10T20:26:23.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ddffb640 10 client.4233.objecter ms_dispatch 0x7fc3e8000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:26:23.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ddffb640 10 client.4233.objecter _finish_command 2 = system:0 2024-09-10T20:26:23.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ddffb640 20 client.4233.objecter put_session s=0x7fc3cc0422c0 osd=4 4 2024-09-10T20:26:23.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ddffb640 15 client.4233.objecter _session_command_op_remove 4 2 2024-09-10T20:26:23.001 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 10 client.4220.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T20:26:23.002 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 15 client.4220.objecter _session_command_op_assign -1 2 2024-09-10T20:26:23.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 20 client.4220.objecter _get_session s=0x7f7f20042350 osd=3 3 2024-09-10T20:26:23.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 20 client.4220.objecter put_session s=0x7f7f20042350 osd=3 3 2024-09-10T20:26:23.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 20 client.4220.objecter _get_session s=0x7f7f20042350 osd=3 3 2024-09-10T20:26:23.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 15 client.4220.objecter _session_command_op_remove -1 2 2024-09-10T20:26:23.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 20 client.4220.objecter get_session s=0x7f7f20042350 osd=3 3 2024-09-10T20:26:23.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 15 client.4220.objecter _session_command_op_assign 3 2 2024-09-10T20:26:23.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 20 client.4220.objecter put_session s=0x7f7f20042350 osd=3 4 2024-09-10T20:26:23.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 10 client.4220.objecter _send_command 2 2024-09-10T20:26:23.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 10 librados: watch_flush enter 2024-09-10T20:26:23.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 10 librados: watch_flush exit 2024-09-10T20:26:23.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 10 client.4233.objecter close_session for osd.4 2024-09-10T20:26:23.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6804/4030285519 conn(0x7fc3cc042970 msgr2=0x7fc3cc044db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 --> v2:172.21.15.176:6808/2953083416 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7f3413fa60 con 0x7f7f20042a00 2024-09-10T20:26:23.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 --2- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6804/4030285519 conn(0x7fc3cc042970 0x7fc3cc044db0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 20 client.4233.objecter put_session s=0x7fc3cc0422c0 osd=4 1 2024-09-10T20:26:23.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 20 client.4233.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 10 client.4233.objecter successfully canceled tick 2024-09-10T20:26:23.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient(hunting): handle_auth_done global_id 4245 payload 274 2024-09-10T20:26:23.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 1 -- 172.21.15.153:0/3375166969 >> v1:172.21.15.176:6789/0 conn(0x7f6c580a41a0 legacy=0x7f6c580a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc3cc03ced0 msgr2=0x7fc3cc03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 --2- 172.21.15.153:0/1384079199 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc3cc03ced0 0x7fc3cc03f390 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3d80ab000 tx=0x7fc3d80a6000 comp rx=0 tx=0).stop 2024-09-10T20:26:23.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 10 monclient: shutdown 2024-09-10T20:26:23.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 1 -- 172.21.15.153:0/3375166969 >> v1:172.21.15.153:6789/0 conn(0x7f6c580a49d0 legacy=0x7f6c5813fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 1 monclient: found mon.noname-a 2024-09-10T20:26:23.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a41a0 msgr2=0x7fc3e8128980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f6c5da10640 1 -- 172.21.15.153:0/3375166969 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c58143450 con 0x7f6c580a4580 2024-09-10T20:26:23.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 --2- 172.21.15.153:0/1384079199 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc3e80a41a0 0x7fc3e8128980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400c410 tx=0x7fc3e400c8d0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3edc65640 1 -- 172.21.15.153:0/1384079199 reap_dead start 2024-09-10T20:26:23.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 shutdown_connections 2024-09-10T20:26:23.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 >> 172.21.15.153:0/1384079199 conn(0x7fc3e80ab7e0 msgr2=0x7fc3e80a03b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 shutdown_connections 2024-09-10T20:26:23.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 1 -- 172.21.15.153:0/1726908079 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7efc700c1460 con 0x7efc800a49d0 2024-09-10T20:26:23.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 1 --2- 172.21.15.153:0/1726908079 >> v2:172.21.15.176:6812/3003473398 conn(0x7efc6403cf20 0x7efc6403f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 1 -- 172.21.15.153:0/1726908079 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7efc700e9ef0 con 0x7efc800a49d0 2024-09-10T20:26:23.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 10 client.4229.objecter ms_dispatch 0x7efc80000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 3 client.4229.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 3 client.4229.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 10 client.4229.objecter checking command 1 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 1 --2- 172.21.15.153:0/1726908079 >> v2:172.21.15.153:6808/1626005750 conn(0x7efc64042a40 0x7efc64044e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 20 client.4229.objecter _get_session s=0x7efc64042390 osd=2 3 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 20 client.4229.objecter put_session s=0x7efc64042390 osd=2 3 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 15 client.4229.objecter _session_command_op_remove -1 1 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 20 client.4229.objecter _get_session s=0x7efc64042390 osd=2 3 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 20 client.4229.objecter get_session s=0x7efc64042390 osd=2 3 2024-09-10T20:26:23.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 15 client.4229.objecter _session_command_op_assign 2 1 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 20 client.4229.objecter put_session s=0x7efc64042390 osd=2 4 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 10 client.4229.objecter _send_command 1 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 1 -- 172.21.15.153:0/1726908079 --> v2:172.21.15.153:6808/1626005750 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efc700a97f0 con 0x7efc64042a40 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 20 client.4229.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 1 -- 172.21.15.153:0/1726908079 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7efc700ec4c0 con 0x7efc800a49d0 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc767fc640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc85f8b640 1 --2- 172.21.15.153:0/1726908079 >> v2:172.21.15.176:6812/3003473398 conn(0x7efc6403cf20 0x7efc6403f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc85f8b640 10 monclient: get_auth_request con 0x7efc6403cf20 auth_method 0 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc8678c640 1 --2- 172.21.15.153:0/1726908079 >> v2:172.21.15.153:6808/1626005750 conn(0x7efc64042a40 0x7efc64044e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7efc8678c640 10 monclient: get_auth_request con 0x7efc64042a40 auth_method 0 2024-09-10T20:26:23.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f1b7fe640 1 -- 172.21.15.153:0/2217099732 <== osd.3 v2:172.21.15.176:6808/2953083416 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f7f2c0a5530 con 0x7f7f20042a00 2024-09-10T20:26:23.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f1b7fe640 10 client.4220.objecter ms_dispatch 0x7f7f34000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:26:23.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f1b7fe640 10 client.4220.objecter _finish_command 2 = system:0 2024-09-10T20:26:23.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f1b7fe640 20 client.4220.objecter put_session s=0x7f7f20042350 osd=3 4 2024-09-10T20:26:23.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f1b7fe640 15 client.4220.objecter _session_command_op_remove 3 2 2024-09-10T20:26:23.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 10 librados: watch_flush enter 2024-09-10T20:26:23.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 10 librados: watch_flush exit 2024-09-10T20:26:23.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 10 client.4220.objecter close_session for osd.3 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6808/2953083416 conn(0x7f7f20042a00 msgr2=0x7f7f20044e40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 --2- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6808/2953083416 conn(0x7f7f20042a00 0x7f7f20044e40 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 20 client.4220.objecter put_session s=0x7f7f20042350 osd=3 1 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 20 client.4220.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 10 client.4220.objecter successfully canceled tick 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7f2003cef0 msgr2=0x7f7f2003f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 --2- 172.21.15.153:0/2217099732 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7f2003cef0 0x7f7f2003f3b0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f7f24004470 tx=0x7f7f24003700 comp rx=0 tx=0).stop 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 10 monclient: shutdown 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 20 monclient: shutdown canceling and discarding version request 1 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a4580 msgr2=0x7f7f34126300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 --2- 172.21.15.153:0/2217099732 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f340a4580 0x7f7f34126300 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f7f2800e870 tx=0x7f7f2800ed30 comp rx=0 tx=0).stop 2024-09-10T20:26:23.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f337fe640 1 -- 172.21.15.153:0/2217099732 reap_dead start 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f1b7fe640 1 -- 172.21.15.153:0/2217099732 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 0 0 0) 0x7f7f28056050 con 0x7f7f340a4580 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f1b7fe640 10 monclient: discarding stray monitor message mon_get_version_reply(handle=1 version=20) 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 shutdown_connections 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 >> 172.21.15.153:0/2217099732 conn(0x7f7f340ab7e0 msgr2=0x7f7f340a1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 shutdown_connections 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 -- 172.21.15.153:0/2217099732 wait complete. 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f7f3969d640 1 librados: shutdown 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 1 -- 172.21.15.153:0/1915786036 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f408401c070 con 0x7f408c1de380 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 1 --2- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6812/3003473398 conn(0x7f407003cf20 0x7f407003f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 1 -- 172.21.15.153:0/1915786036 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f408404f0e0 con 0x7f408c1de380 2024-09-10T20:26:23.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 10 client.4226.objecter ms_dispatch 0x7f408c000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 3 client.4226.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 3 client.4226.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 10 client.4226.objecter checking command 1 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 1 --2- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6800/2372911183 conn(0x7f4070042a30 0x7f4070044e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 20 client.4226.objecter _get_session s=0x7f4070042380 osd=5 3 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 20 client.4226.objecter put_session s=0x7f4070042380 osd=5 3 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 15 client.4226.objecter _session_command_op_remove -1 1 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 20 client.4226.objecter _get_session s=0x7f4070042380 osd=5 3 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 20 client.4226.objecter get_session s=0x7f4070042380 osd=5 3 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 15 client.4226.objecter _session_command_op_assign 5 1 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 20 client.4226.objecter put_session s=0x7f4070042380 osd=5 4 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 10 client.4226.objecter _send_command 1 2024-09-10T20:26:23.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 1 -- 172.21.15.153:0/1915786036 --> v2:172.21.15.176:6800/2372911183 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4070045500 con 0x7f4070042a30 2024-09-10T20:26:23.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 20 client.4226.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4091d3a640 1 --2- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6812/3003473398 conn(0x7f407003cf20 0x7f407003f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4091d3a640 10 monclient: get_auth_request con 0x7f407003cf20 auth_method 0 2024-09-10T20:26:23.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4091539640 1 --2- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6800/2372911183 conn(0x7f4070042a30 0x7f4070044e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f4091539640 10 monclient: get_auth_request con 0x7f4070042a30 auth_method 0 2024-09-10T20:26:23.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 1 -- 172.21.15.153:0/1915786036 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f4084056050 con 0x7f408c1de380 2024-09-10T20:26:23.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7f40827fc640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-10T20:26:23.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 -- 172.21.15.153:0/1384079199 wait complete. 2024-09-10T20:26:23.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.926+0000 7fc3ef6c8640 1 librados: shutdown 2024-09-10T20:26:23.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f4091d3a640 1 --2- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6812/3003473398 conn(0x7f407003cf20 0x7f407003f3e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4088004e30 tx=0x7f4088009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f40827fc640 10 client.4226.objecter ms_handle_connect 0x7f407003cf20 2024-09-10T20:26:23.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7efc767fc640 10 client.4229.objecter ms_handle_connect 0x7efc6403cf20 2024-09-10T20:26:23.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7efc85f8b640 1 --2- 172.21.15.153:0/1726908079 >> v2:172.21.15.176:6812/3003473398 conn(0x7efc6403cf20 0x7efc6403f3e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc7c007ca0 tx=0x7efc7c007bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient: get_monmap_and_config 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient: monmap: 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:22.935767+0000 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:22.935767+0000 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:23.020 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:23.021 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:23.021 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:23.021 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:23.021 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5da10640 1 -- 172.21.15.153:0/3375166969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a4580 msgr2=0x7f6c5813d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5da10640 1 --2- 172.21.15.153:0/3375166969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a4580 0x7f6c5813d7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6c480097c0 tx=0x7f6c4801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41917d85e74ae2f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 1 -- 172.21.15.153:0/3375166969 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c480b0070 con 0x7f6c580a4580 2024-09-10T20:26:23.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 10 monclient: dump: 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 1 Processor -- start 2024-09-10T20:26:23.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 1 -- 172.21.15.153:0/3375166969 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c480a97f0 con 0x7f6c580a4580 2024-09-10T20:26:23.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 5 monclient: authenticate success, global_id 4245 2024-09-10T20:26:23.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:23.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 4 set_mon_vals no callback set 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 1 -- 172.21.15.153:0/3375166969 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c480a9bc0 con 0x7f6c580a4580 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c5d20f640 10 monclient: dump: 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.024 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.025 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.025 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.025 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 1 -- start start 2024-09-10T20:26:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient: init 2024-09-10T20:26:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient: _add_conns ranks=[2,4,1,0,3] 2024-09-10T20:26:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac880a4950 0x7fac880a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient(hunting): picked mon.noname-b con 0x7fac880a4950 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fac880a5270 0x7fac8813a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient(hunting): picked mon.noname-c con 0x7fac880a5270 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:23.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fac8813ac60 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:23.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac880a3160 con 0x7fac8813ac60 2024-09-10T20:26:23.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fac880a0e40 con 0x7fac880a5270 2024-09-10T20:26:23.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fac880a21c0 con 0x7fac880a4950 2024-09-10T20:26:23.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac8da60640 10 monclient(hunting): authenticate will time out at 943.075538s 2024-09-10T20:26:23.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 10 monclient: shutdown 2024-09-10T20:26:23.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 1 -- 172.21.15.153:0/3375166969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a4580 msgr2=0x7f6c5813d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 1 --2- 172.21.15.153:0/3375166969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a4580 0x7f6c5813d7c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6c480097c0 tx=0x7f6c4801c360 comp rx=0 tx=0).stop 2024-09-10T20:26:23.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac880a4950 0x7fac880a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac880a4950 0x7fac880a4d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55570/0 (socket says 172.21.15.153:55570) 2024-09-10T20:26:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 1 -- 172.21.15.153:0/2966764625 learned_addr learned my addr 172.21.15.153:0/2966764625 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 10 monclient(hunting): get_auth_request con 0x7fac880a4950 auth_method 0 2024-09-10T20:26:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f4091539640 1 --2- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6800/2372911183 conn(0x7f4070042a30 0x7f4070044e70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f40827fc640 10 client.4226.objecter ms_handle_connect 0x7f4070042a30 2024-09-10T20:26:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7efc8678c640 1 --2- 172.21.15.153:0/1726908079 >> v2:172.21.15.153:6808/1626005750 conn(0x7efc64042a40 0x7efc64044e80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7efc767fc640 10 client.4229.objecter ms_handle_connect 0x7efc64042a40 2024-09-10T20:26:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac86ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 1 -- 172.21.15.153:0/3375166969 shutdown_connections 2024-09-10T20:26:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 1 --2- 172.21.15.153:0/3375166969 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a4580 0x7f6c5813d7c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7f6c60475640 1 -- 172.21.15.153:0/3375166969 >> 172.21.15.153:0/3375166969 conn(0x7f6c580ab7e0 msgr2=0x7f6c580a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac867fc640 1 --2- 172.21.15.153:0/2966764625 >> v2:172.21.15.176:3300/0 conn(0x7fac880a5270 0x7fac8813a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac867fc640 10 monclient(hunting): get_auth_request con 0x7fac880a5270 auth_method 0 2024-09-10T20:26:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac867fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.930+0000 7fac867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 -- 172.21.15.153:0/3375166969 shutdown_connections 2024-09-10T20:26:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7efc767fc640 1 -- 172.21.15.153:0/1726908079 <== osd.2 v2:172.21.15.153:6808/1626005750 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efc700a97f0 con 0x7efc64042a40 2024-09-10T20:26:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7efc767fc640 10 client.4229.objecter ms_dispatch 0x7efc80000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:26:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7efc767fc640 10 client.4229.objecter _finish_command 1 = system:0 2024-09-10T20:26:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7efc767fc640 20 client.4229.objecter put_session s=0x7efc64042390 osd=2 4 2024-09-10T20:26:23.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7efc767fc640 15 client.4229.objecter _session_command_op_remove 2 1 2024-09-10T20:26:23.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 -- 172.21.15.153:0/3375166969 wait complete. 2024-09-10T20:26:23.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient: monmap: 2024-09-10T20:26:23.034 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.034 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:23.034 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:23.034 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 librados: starting msgr at 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 librados: starting objecter 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 Processor -- start 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 -- start start 2024-09-10T20:26:23.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 librados: setting wanted keys 2024-09-10T20:26:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 librados: calling monclient init 2024-09-10T20:26:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient: init 2024-09-10T20:26:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:26:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f40827fc640 1 -- 172.21.15.153:0/1915786036 <== osd.5 v2:172.21.15.176:6800/2372911183 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4070045500 con 0x7f4070042a30 2024-09-10T20:26:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f40827fc640 10 client.4226.objecter ms_dispatch 0x7f408c000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:26:23.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f40827fc640 10 client.4226.objecter _finish_command 1 = system:0 2024-09-10T20:26:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f40827fc640 20 client.4226.objecter put_session s=0x7f4070042380 osd=5 4 2024-09-10T20:26:23.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f40827fc640 15 client.4226.objecter _session_command_op_remove 5 1 2024-09-10T20:26:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a41a0 0x7f6c5813f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient(hunting): picked mon.noname-a con 0x7f6c580a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c580a4580 0x7f6c581377a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient(hunting): picked mon.noname-c con 0x7f6c580a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c580a49d0 0x7f6c581d0250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient(hunting): picked mon.noname-b con 0x7f6c580a49d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.038 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6c58144040 con 0x7f6c580a41a0 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6c58070400 con 0x7f6c580a49d0 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6c581441c0 con 0x7f6c580a4580 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c60475640 10 monclient(hunting): authenticate will time out at 943.077691s 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5e211640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a41a0 0x7f6c5813f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5e211640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a41a0 0x7f6c5813f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42262/0 (socket says 172.21.15.153:42262) 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5e211640 1 -- 172.21.15.153:0/1635566884 learned_addr learned my addr 172.21.15.153:0/1635566884 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5e211640 10 monclient(hunting): get_auth_request con 0x7f6c580a41a0 auth_method 0 2024-09-10T20:26:23.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5e211640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5e211640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5e211640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 1 --2- 172.21.15.153:0/1635566884 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c580a4580 0x7f6c581377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient(hunting): get_auth_request con 0x7f6c580a4580 auth_method 0 2024-09-10T20:26:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7fac86ffd640 10 monclient(hunting): handle_auth_done global_id 4232 payload 274 2024-09-10T20:26:23.040 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7fac86ffd640 1 -- 172.21.15.153:0/2966764625 >> v2:172.21.15.176:3300/0 conn(0x7fac880a5270 msgr2=0x7fac8813a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5ea12640 1 --2- 172.21.15.153:0/1635566884 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c580a49d0 0x7f6c581d0250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5ea12640 10 monclient(hunting): get_auth_request con 0x7f6c580a49d0 auth_method 0 2024-09-10T20:26:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5ea12640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5ea12640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5ea12640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.041 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7fac86ffd640 1 --2- 172.21.15.153:0/2966764625 >> v2:172.21.15.176:3300/0 conn(0x7fac880a5270 0x7fac8813a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7fac86ffd640 1 -- 172.21.15.153:0/2966764625 >> v1:172.21.15.176:6789/0 conn(0x7fac8813ac60 legacy=0x7fac8813d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7fac86ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7fac86ffd640 1 monclient: found mon.noname-b 2024-09-10T20:26:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7fac86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7fac86ffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7fac86ffd640 1 -- 172.21.15.153:0/2966764625 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac880a19e0 con 0x7fac880a4950 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5ea12640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5ea12640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5ea12640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient(hunting): handle_auth_done global_id 4235 payload 824 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 1 -- 172.21.15.153:0/1635566884 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c580a49d0 msgr2=0x7f6c581d0250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 1 --2- 172.21.15.153:0/1635566884 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6c580a49d0 0x7f6c581d0250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 1 -- 172.21.15.153:0/1635566884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a41a0 msgr2=0x7f6c5813f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 1 --2- 172.21.15.153:0/1635566884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a41a0 0x7f6c5813f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 1 monclient: found mon.noname-c 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5da10640 1 -- 172.21.15.153:0/1635566884 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c581400c0 con 0x7f6c580a4580 2024-09-10T20:26:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.934+0000 7f6c5e211640 1 --2- 172.21.15.153:0/1635566884 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6c580a41a0 0x7f6c5813f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:26:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c5da10640 1 -- 172.21.15.153:0/1635566884 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c580a4580 msgr2=0x7f6c581377a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c5da10640 1 --2- 172.21.15.153:0/1635566884 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c580a4580 0x7f6c581377a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c480ad000 tx=0x7f6c48019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 client.?.objecter ms_handle_connect 0x7f6c580a4580 2024-09-10T20:26:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac86ffd640 1 -- 172.21.15.153:0/2966764625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac880a4950 msgr2=0x7fac880a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac86ffd640 1 --2- 172.21.15.153:0/2966764625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac880a4950 0x7fac880a4d30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009870 tx=0x7fac7c01c210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb49daf2ce67252 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 1 -- 172.21.15.153:0/1635566884 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c480b9070 con 0x7f6c580a4580 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 monclient: dump: 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.048 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 1 -- 172.21.15.153:0/1635566884 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c480b5ac0 con 0x7f6c580a4580 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 1 -- 172.21.15.153:0/1635566884 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c480b0040 con 0x7f6c580a4580 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4effd640 10 monclient: dump: 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.049 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.050 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c4ffff640 4 set_mon_vals no callback set 2024-09-10T20:26:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 1 -- 172.21.15.153:0/2966764625 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0b0070 con 0x7fac880a4950 2024-09-10T20:26:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 10 monclient: dump: 2024-09-10T20:26:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.051 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 1 -- 172.21.15.153:0/2966764625 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac7c0ab600 con 0x7fac880a4950 2024-09-10T20:26:23.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 1 -- 172.21.15.153:0/2966764625 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac7c0ab9d0 con 0x7fac880a4950 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 5 monclient: authenticate success, global_id 4232 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 4 set_mon_vals no callback set 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac85ffb640 10 monclient: dump: 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.053 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 10 monclient: shutdown 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 1 -- 172.21.15.153:0/2966764625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac880a4950 msgr2=0x7fac880a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 1 --2- 172.21.15.153:0/2966764625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac880a4950 0x7fac880a4d30 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fac7c009870 tx=0x7fac7c01c210 comp rx=0 tx=0).stop 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 1 -- 172.21.15.153:0/2966764625 shutdown_connections 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 1 --2- 172.21.15.153:0/2966764625 >> v2:172.21.15.176:3300/0 conn(0x7fac880a5270 0x7fac8813a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 1 --2- 172.21.15.153:0/2966764625 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac880a4950 0x7fac880a4d30 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7fac8da60640 1 -- 172.21.15.153:0/2966764625 >> 172.21.15.153:0/2966764625 conn(0x7fac880a96f0 msgr2=0x7fac880a5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c60475640 5 monclient: authenticate success, global_id 4235 2024-09-10T20:26:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c60475640 10 monclient: _renew_subs 2024-09-10T20:26:23.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c60475640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c5813ebd0 con 0x7f6c580a4580 2024-09-10T20:26:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c60475640 10 client.4235.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c60475640 10 monclient: _renew_subs 2024-09-10T20:26:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c60475640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c58140970 con 0x7f6c580a4580 2024-09-10T20:26:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.938+0000 7f6c60475640 1 librados: init done 2024-09-10T20:26:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7f6c4effd640 1 -- 172.21.15.153:0/1635566884 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f6c480bc070 con 0x7f6c580a4580 2024-09-10T20:26:23.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 -- 172.21.15.153:0/2966764625 shutdown_connections 2024-09-10T20:26:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7f6c4effd640 1 --2- 172.21.15.153:0/1635566884 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6c3c03cea0 0x7f6c3c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7f6c4effd640 1 -- 172.21.15.153:0/1635566884 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f6c480e99a0 con 0x7f6c580a4580 2024-09-10T20:26:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7f6c4effd640 10 client.4235.objecter ms_dispatch 0x7f6c58000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7f6c4effd640 3 client.4235.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7f6c4effd640 3 client.4235.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7f6c4effd640 20 client.4235.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 -- 172.21.15.153:0/2966764625 wait complete. 2024-09-10T20:26:23.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7f6c5e211640 1 --2- 172.21.15.153:0/1635566884 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6c3c03cea0 0x7f6c3c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7f6c5e211640 10 monclient: get_auth_request con 0x7f6c3c03cea0 auth_method 0 2024-09-10T20:26:23.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient: monmap: 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 librados: starting msgr at 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 librados: starting objecter 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 Processor -- start 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 -- start start 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 librados: setting wanted keys 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 librados: calling monclient init 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient: init 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac880a4950 0x7fac8813d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient(hunting): picked mon.noname-b con 0x7fac880a4950 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac880a5270 0x7fac88138ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient(hunting): picked mon.noname-a con 0x7fac880a5270 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8813ac60 0x7fac88139010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient(hunting): picked mon.noname-c con 0x7fac8813ac60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fac880a5650 con 0x7fac880a5270 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fac880a6400 con 0x7fac880a4950 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fac880709b0 con 0x7fac8813ac60 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac8da60640 10 monclient(hunting): authenticate will time out at 943.087711s 2024-09-10T20:26:23.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac867fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac880a5270 0x7fac88138ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac867fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac880a5270 0x7fac88138ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42272/0 (socket says 172.21.15.153:42272) 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac867fc640 1 -- 172.21.15.153:0/2754551641 learned_addr learned my addr 172.21.15.153:0/2754551641 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac867fc640 10 monclient(hunting): get_auth_request con 0x7fac880a5270 auth_method 0 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac867fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac867fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.942+0000 7fac867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac877fe640 1 --2- 172.21.15.153:0/2754551641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8813ac60 0x7fac88139010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac877fe640 10 monclient(hunting): get_auth_request con 0x7fac8813ac60 auth_method 0 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac877fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac877fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac877fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac877fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac877fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac877fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 10 monclient(hunting): handle_auth_done global_id 4251 payload 824 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 1 -- 172.21.15.153:0/2754551641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8813ac60 msgr2=0x7fac88139010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 1 --2- 172.21.15.153:0/2754551641 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fac8813ac60 0x7fac88139010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 1 -- 172.21.15.153:0/2754551641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac880a4950 msgr2=0x7fac8813d2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 1 --2- 172.21.15.153:0/2754551641 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fac880a4950 0x7fac8813d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 1 monclient: found mon.noname-a 2024-09-10T20:26:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 1 -- 172.21.15.153:0/2754551641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac88071560 con 0x7fac880a5270 2024-09-10T20:26:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7f6c5e211640 1 --2- 172.21.15.153:0/1635566884 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6c3c03cea0 0x7f6c3c03f360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6c54003f80 tx=0x7f6c54009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7f6c4effd640 10 client.4235.objecter ms_handle_connect 0x7f6c3c03cea0 2024-09-10T20:26:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 1 -- 172.21.15.153:0/2754551641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac880a5270 msgr2=0x7fac88138ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac867fc640 1 --2- 172.21.15.153:0/2754551641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac880a5270 0x7fac88138ad0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fac7800d7d0 tx=0x7fac7800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 client.?.objecter ms_handle_connect 0x7fac880a5270 2024-09-10T20:26:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 1 -- 172.21.15.153:0/2754551641 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac78004020 con 0x7fac880a5270 2024-09-10T20:26:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 monclient: dump: 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.068 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 1 -- 172.21.15.153:0/2754551641 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac780041c0 con 0x7fac880a5270 2024-09-10T20:26:23.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 1 -- 172.21.15.153:0/2754551641 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac780043b0 con 0x7fac880a5270 2024-09-10T20:26:23.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac6ffff640 10 monclient: dump: 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.070 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac8da60640 5 monclient: authenticate success, global_id 4251 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac8da60640 10 monclient: _renew_subs 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac8da60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac880a0130 con 0x7fac880a5270 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac857fa640 4 set_mon_vals no callback set 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac8da60640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac8da60640 10 monclient: _renew_subs 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac8da60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac8806f1b0 con 0x7fac880a5270 2024-09-10T20:26:23.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.946+0000 7fac8da60640 1 librados: init done 2024-09-10T20:26:23.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 10 client.4235.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 15 client.4235.objecter _session_command_op_assign -1 1 2024-09-10T20:26:23.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 1 --2- 172.21.15.153:0/1635566884 >> v2:172.21.15.153:6800/4216589932 conn(0x7f6c5813d640 0x7f6c581d0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 20 client.4235.objecter _get_session s=0x7f6c58140bb0 osd=0 3 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 20 client.4235.objecter put_session s=0x7f6c58140bb0 osd=0 3 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 20 client.4235.objecter _get_session s=0x7f6c58140bb0 osd=0 3 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 15 client.4235.objecter _session_command_op_remove -1 1 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 20 client.4235.objecter get_session s=0x7f6c58140bb0 osd=0 3 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac8da60640 10 client.4251.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac8da60640 15 client.4251.objecter _session_command_op_assign -1 1 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac8da60640 20 client.4251.objecter _get_session osd=-1 returning homeless 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac8da60640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac8da60640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fac880a4d30 con 0x7fac880a5270 2024-09-10T20:26:23.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 15 client.4235.objecter _session_command_op_assign 0 1 2024-09-10T20:26:23.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 20 client.4235.objecter put_session s=0x7f6c58140bb0 osd=0 4 2024-09-10T20:26:23.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 10 client.4235.objecter _send_command 1 2024-09-10T20:26:23.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c5ea12640 1 --2- 172.21.15.153:0/1635566884 >> v2:172.21.15.153:6800/4216589932 conn(0x7f6c5813d640 0x7f6c581d0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c5ea12640 10 monclient: get_auth_request con 0x7f6c5813d640 auth_method 0 2024-09-10T20:26:23.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 --> v2:172.21.15.153:6800/4216589932 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6c580a3720 con 0x7f6c5813d640 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 1 -- 172.21.15.153:0/2754551641 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fac7801b020 con 0x7fac880a5270 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 1 --2- 172.21.15.153:0/2754551641 >> v2:172.21.15.176:6812/3003473398 conn(0x7fac7403cef0 0x7fac7403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 1 -- 172.21.15.153:0/2754551641 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7fac7804fa80 con 0x7fac880a5270 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 10 client.4251.objecter ms_dispatch 0x7fac88000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 3 client.4251.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 3 client.4251.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 10 client.4251.objecter checking command 1 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 1 --2- 172.21.15.153:0/2754551641 >> v2:172.21.15.153:6804/1715391555 conn(0x7fac74042a80 0x7fac74044ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 20 client.4251.objecter _get_session s=0x7fac740423d0 osd=1 3 2024-09-10T20:26:23.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 20 client.4251.objecter put_session s=0x7fac740423d0 osd=1 3 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 15 client.4251.objecter _session_command_op_remove -1 1 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 20 client.4251.objecter _get_session s=0x7fac740423d0 osd=1 3 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 20 client.4251.objecter get_session s=0x7fac740423d0 osd=1 3 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 15 client.4251.objecter _session_command_op_assign 1 1 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 20 client.4251.objecter put_session s=0x7fac740423d0 osd=1 4 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 10 client.4251.objecter _send_command 1 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 1 -- 172.21.15.153:0/2754551641 --> v2:172.21.15.153:6804/1715391555 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fac78008e50 con 0x7fac74042a80 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 20 client.4251.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 1 -- 172.21.15.153:0/2754551641 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fac7804fd60 con 0x7fac880a5270 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac6ffff640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac877fe640 1 --2- 172.21.15.153:0/2754551641 >> v2:172.21.15.153:6804/1715391555 conn(0x7fac74042a80 0x7fac74044ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac877fe640 10 monclient: get_auth_request con 0x7fac74042a80 auth_method 0 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac86ffd640 1 --2- 172.21.15.153:0/2754551641 >> v2:172.21.15.176:6812/3003473398 conn(0x7fac7403cef0 0x7fac7403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7fac86ffd640 10 monclient: get_auth_request con 0x7fac7403cef0 auth_method 0 2024-09-10T20:26:23.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 10 client.4229.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T20:26:23.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 15 client.4229.objecter _session_command_op_assign -1 2 2024-09-10T20:26:23.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 20 client.4229.objecter _get_session s=0x7efc64042390 osd=2 3 2024-09-10T20:26:23.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 20 client.4229.objecter put_session s=0x7efc64042390 osd=2 3 2024-09-10T20:26:23.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 20 client.4229.objecter _get_session s=0x7efc64042390 osd=2 3 2024-09-10T20:26:23.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 15 client.4229.objecter _session_command_op_remove -1 2 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 20 client.4229.objecter get_session s=0x7efc64042390 osd=2 3 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 15 client.4229.objecter _session_command_op_assign 2 2 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 20 client.4229.objecter put_session s=0x7efc64042390 osd=2 4 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 10 client.4229.objecter _send_command 2 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 --> v2:172.21.15.153:6808/1626005750 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efc8013f9f0 con 0x7efc64042a40 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc767fc640 1 -- 172.21.15.153:0/1726908079 <== osd.2 v2:172.21.15.153:6808/1626005750 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efc8013f9f0 con 0x7efc64042a40 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc767fc640 10 client.4229.objecter ms_dispatch 0x7efc80000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc767fc640 10 client.4229.objecter _finish_command 2 = system:0 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc767fc640 20 client.4229.objecter put_session s=0x7efc64042390 osd=2 4 2024-09-10T20:26:23.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc767fc640 15 client.4229.objecter _session_command_op_remove 2 2 2024-09-10T20:26:23.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 10 librados: watch_flush enter 2024-09-10T20:26:23.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 10 librados: watch_flush exit 2024-09-10T20:26:23.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 10 client.4229.objecter close_session for osd.2 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 >> v2:172.21.15.153:6808/1626005750 conn(0x7efc64042a40 msgr2=0x7efc64044e80 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 --2- 172.21.15.153:0/1726908079 >> v2:172.21.15.153:6808/1626005750 conn(0x7efc64042a40 0x7efc64044e80 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 20 client.4229.objecter put_session s=0x7efc64042390 osd=2 1 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 20 client.4229.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 10 client.4229.objecter successfully canceled tick 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 >> v2:172.21.15.176:6812/3003473398 conn(0x7efc6403cf20 msgr2=0x7efc6403f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 --2- 172.21.15.153:0/1726908079 >> v2:172.21.15.176:6812/3003473398 conn(0x7efc6403cf20 0x7efc6403f3e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7efc7c007ca0 tx=0x7efc7c007bf0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 10 monclient: shutdown 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc800a49d0 msgr2=0x7efc80132b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 --2- 172.21.15.153:0/1726908079 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7efc800a49d0 0x7efc80132b70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efc700ae040 tx=0x7efc7001aba0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc8678c640 1 -- 172.21.15.153:0/1726908079 reap_dead start 2024-09-10T20:26:23.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 shutdown_connections 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 >> 172.21.15.153:0/1726908079 conn(0x7efc800ab7e0 msgr2=0x7efc80140a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 shutdown_connections 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 -- 172.21.15.153:0/1726908079 wait complete. 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.950+0000 7efc881ef640 1 librados: shutdown 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 10 client.4226.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 15 client.4226.objecter _session_command_op_assign -1 2 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 20 client.4226.objecter _get_session s=0x7f4070042380 osd=5 3 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 20 client.4226.objecter put_session s=0x7f4070042380 osd=5 3 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 20 client.4226.objecter _get_session s=0x7f4070042380 osd=5 3 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 15 client.4226.objecter _session_command_op_remove -1 2 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 20 client.4226.objecter get_session s=0x7f4070042380 osd=5 3 2024-09-10T20:26:23.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 15 client.4226.objecter _session_command_op_assign 5 2 2024-09-10T20:26:23.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 20 client.4226.objecter put_session s=0x7f4070042380 osd=5 4 2024-09-10T20:26:23.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 10 client.4226.objecter _send_command 2 2024-09-10T20:26:23.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 --> v2:172.21.15.176:6800/2372911183 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f408c0a2060 con 0x7f4070042a30 2024-09-10T20:26:23.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7fac6ffff640 10 client.4251.objecter ms_handle_connect 0x7fac7403cef0 2024-09-10T20:26:23.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7fac86ffd640 1 --2- 172.21.15.153:0/2754551641 >> v2:172.21.15.176:6812/3003473398 conn(0x7fac7403cef0 0x7fac7403f3b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0ab000 tx=0x7fac7c004740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f6c4effd640 10 client.4235.objecter ms_handle_connect 0x7f6c5813d640 2024-09-10T20:26:23.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f6c5ea12640 1 --2- 172.21.15.153:0/1635566884 >> v2:172.21.15.153:6800/4216589932 conn(0x7f6c5813d640 0x7f6c581d0ca0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f40827fc640 1 -- 172.21.15.153:0/1915786036 <== osd.5 v2:172.21.15.176:6800/2372911183 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f408c0a2060 con 0x7f4070042a30 2024-09-10T20:26:23.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f40827fc640 10 client.4226.objecter ms_dispatch 0x7f408c000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:26:23.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f40827fc640 10 client.4226.objecter _finish_command 2 = system:0 2024-09-10T20:26:23.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f40827fc640 20 client.4226.objecter put_session s=0x7f4070042380 osd=5 4 2024-09-10T20:26:23.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f40827fc640 15 client.4226.objecter _session_command_op_remove 5 2 2024-09-10T20:26:23.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f6c4effd640 1 -- 172.21.15.153:0/1635566884 <== osd.0 v2:172.21.15.153:6800/4216589932 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6c580a3720 con 0x7f6c5813d640 2024-09-10T20:26:23.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f6c4effd640 10 client.4235.objecter ms_dispatch 0x7f6c58000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:26:23.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f6c4effd640 10 client.4235.objecter _finish_command 1 = system:0 2024-09-10T20:26:23.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f6c4effd640 20 client.4235.objecter put_session s=0x7f6c58140bb0 osd=0 4 2024-09-10T20:26:23.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f6c4effd640 15 client.4235.objecter _session_command_op_remove 0 1 2024-09-10T20:26:23.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 10 librados: watch_flush enter 2024-09-10T20:26:23.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 10 librados: watch_flush exit 2024-09-10T20:26:23.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 10 client.4226.objecter close_session for osd.5 2024-09-10T20:26:23.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6800/2372911183 conn(0x7f4070042a30 msgr2=0x7f4070044e70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 --2- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6800/2372911183 conn(0x7f4070042a30 0x7f4070044e70 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.086 INFO:teuthology.orchestra.run.smithi153.stdout:51539607557 2024-09-10T20:26:23.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 20 client.4226.objecter put_session s=0x7f4070042380 osd=5 1 2024-09-10T20:26:23.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 20 client.4226.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 10 client.4226.objecter successfully canceled tick 2024-09-10T20:26:23.087 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-10T20:26:23.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7fac877fe640 1 --2- 172.21.15.153:0/2754551641 >> v2:172.21.15.153:6804/1715391555 conn(0x7fac74042a80 0x7fac74044ec0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7fac6ffff640 10 client.4251.objecter ms_handle_connect 0x7fac74042a80 2024-09-10T20:26:23.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6812/3003473398 conn(0x7f407003cf20 msgr2=0x7f407003f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 --2- 172.21.15.153:0/1915786036 >> v2:172.21.15.176:6812/3003473398 conn(0x7f407003cf20 0x7f407003f3e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4088004e30 tx=0x7f4088009040 comp rx=0 tx=0).stop 2024-09-10T20:26:23.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 10 monclient: shutdown 2024-09-10T20:26:23.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c1de380 msgr2=0x7f408c091df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 --2- 172.21.15.153:0/1915786036 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f408c1de380 0x7f408c091df0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f408400b9e0 tx=0x7f408400bea0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f409253b640 1 -- 172.21.15.153:0/1915786036 reap_dead start 2024-09-10T20:26:23.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 shutdown_connections 2024-09-10T20:26:23.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 >> 172.21.15.153:0/1915786036 conn(0x7f408c0a7830 msgr2=0x7f408c09d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 shutdown_connections 2024-09-10T20:26:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 -- 172.21.15.153:0/1915786036 wait complete. 2024-09-10T20:26:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7f4093f9e640 1 librados: shutdown 2024-09-10T20:26:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7fac6ffff640 1 -- 172.21.15.153:0/2754551641 <== osd.1 v2:172.21.15.153:6804/1715391555 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fac78008e50 con 0x7fac74042a80 2024-09-10T20:26:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7fac6ffff640 10 client.4251.objecter ms_dispatch 0x7fac88000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:26:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7fac6ffff640 10 client.4251.objecter _finish_command 1 = system:0 2024-09-10T20:26:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7fac6ffff640 20 client.4251.objecter put_session s=0x7fac740423d0 osd=1 4 2024-09-10T20:26:23.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.954+0000 7fac6ffff640 15 client.4251.objecter _session_command_op_remove 1 1 2024-09-10T20:26:23.091 INFO:teuthology.orchestra.run.smithi153.stdout:51539607557 2024-09-10T20:26:23.091 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-10T20:26:23.092 INFO:teuthology.orchestra.run.smithi153.stdout:55834574852 2024-09-10T20:26:23.092 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-10T20:26:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 10 client.4235.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T20:26:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 15 client.4235.objecter _session_command_op_assign -1 2 2024-09-10T20:26:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 20 client.4235.objecter _get_session s=0x7f6c58140bb0 osd=0 3 2024-09-10T20:26:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 20 client.4235.objecter put_session s=0x7f6c58140bb0 osd=0 3 2024-09-10T20:26:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 20 client.4235.objecter _get_session s=0x7f6c58140bb0 osd=0 3 2024-09-10T20:26:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 15 client.4235.objecter _session_command_op_remove -1 2 2024-09-10T20:26:23.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 20 client.4235.objecter get_session s=0x7f6c58140bb0 osd=0 3 2024-09-10T20:26:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 15 client.4235.objecter _session_command_op_assign 0 2 2024-09-10T20:26:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 20 client.4235.objecter put_session s=0x7f6c58140bb0 osd=0 4 2024-09-10T20:26:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 10 client.4235.objecter _send_command 2 2024-09-10T20:26:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 --> v2:172.21.15.153:6800/4216589932 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6c58137ce0 con 0x7f6c5813d640 2024-09-10T20:26:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c4effd640 1 -- 172.21.15.153:0/1635566884 <== osd.0 v2:172.21.15.153:6800/4216589932 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6c58137ce0 con 0x7f6c5813d640 2024-09-10T20:26:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c4effd640 10 client.4235.objecter ms_dispatch 0x7f6c58000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:26:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c4effd640 10 client.4235.objecter _finish_command 2 = system:0 2024-09-10T20:26:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c4effd640 20 client.4235.objecter put_session s=0x7f6c58140bb0 osd=0 4 2024-09-10T20:26:23.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c4effd640 15 client.4235.objecter _session_command_op_remove 0 2 2024-09-10T20:26:23.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 10 librados: watch_flush enter 2024-09-10T20:26:23.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 10 librados: watch_flush exit 2024-09-10T20:26:23.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 10 client.4235.objecter close_session for osd.0 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 >> v2:172.21.15.153:6800/4216589932 conn(0x7f6c5813d640 msgr2=0x7f6c581d0ca0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 --2- 172.21.15.153:0/1635566884 >> v2:172.21.15.153:6800/4216589932 conn(0x7f6c5813d640 0x7f6c581d0ca0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 20 client.4235.objecter put_session s=0x7f6c58140bb0 osd=0 1 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 20 client.4235.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 10 client.4235.objecter successfully canceled tick 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6c3c03cea0 msgr2=0x7f6c3c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 --2- 172.21.15.153:0/1635566884 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6c3c03cea0 0x7f6c3c03f360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6c54003f80 tx=0x7f6c54009040 comp rx=0 tx=0).stop 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 10 monclient: shutdown 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c580a4580 msgr2=0x7f6c581377a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 --2- 172.21.15.153:0/1635566884 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6c580a4580 0x7f6c581377a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6c480ad000 tx=0x7f6c48019100 comp rx=0 tx=0).stop 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c5ea12640 1 -- 172.21.15.153:0/1635566884 reap_dead start 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 shutdown_connections 2024-09-10T20:26:23.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 >> 172.21.15.153:0/1635566884 conn(0x7f6c580ab7e0 msgr2=0x7f6c5813bd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 shutdown_connections 2024-09-10T20:26:23.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 -- 172.21.15.153:0/1635566884 wait complete. 2024-09-10T20:26:23.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.978+0000 7f6c60475640 1 librados: shutdown 2024-09-10T20:26:23.097 INFO:teuthology.orchestra.run.smithi153.stdout:47244640261 2024-09-10T20:26:23.098 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-10T20:26:23.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 10 client.4251.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-10T20:26:23.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 15 client.4251.objecter _session_command_op_assign -1 2 2024-09-10T20:26:23.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 20 client.4251.objecter _get_session s=0x7fac740423d0 osd=1 3 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 20 client.4251.objecter put_session s=0x7fac740423d0 osd=1 3 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 20 client.4251.objecter _get_session s=0x7fac740423d0 osd=1 3 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 15 client.4251.objecter _session_command_op_remove -1 2 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 20 client.4251.objecter get_session s=0x7fac740423d0 osd=1 3 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 15 client.4251.objecter _session_command_op_assign 1 2 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 20 client.4251.objecter put_session s=0x7fac740423d0 osd=1 4 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 10 client.4251.objecter _send_command 2 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 --> v2:172.21.15.153:6804/1715391555 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fac88133f30 con 0x7fac74042a80 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac6ffff640 1 -- 172.21.15.153:0/2754551641 <== osd.1 v2:172.21.15.153:6804/1715391555 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fac88133f30 con 0x7fac74042a80 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac6ffff640 10 client.4251.objecter ms_dispatch 0x7fac88000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac6ffff640 10 client.4251.objecter _finish_command 2 = system:0 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac6ffff640 20 client.4251.objecter put_session s=0x7fac740423d0 osd=1 4 2024-09-10T20:26:23.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac6ffff640 15 client.4251.objecter _session_command_op_remove 1 2 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 10 librados: watch_flush enter 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 10 librados: watch_flush exit 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 10 client.4251.objecter close_session for osd.1 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 >> v2:172.21.15.153:6804/1715391555 conn(0x7fac74042a80 msgr2=0x7fac74044ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 --2- 172.21.15.153:0/2754551641 >> v2:172.21.15.153:6804/1715391555 conn(0x7fac74042a80 0x7fac74044ec0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 20 client.4251.objecter put_session s=0x7fac740423d0 osd=1 1 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 20 client.4251.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 10 client.4251.objecter successfully canceled tick 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 >> v2:172.21.15.176:6812/3003473398 conn(0x7fac7403cef0 msgr2=0x7fac7403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 --2- 172.21.15.153:0/2754551641 >> v2:172.21.15.176:6812/3003473398 conn(0x7fac7403cef0 0x7fac7403f3b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fac7c0ab000 tx=0x7fac7c004740 comp rx=0 tx=0).stop 2024-09-10T20:26:23.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 10 monclient: shutdown 2024-09-10T20:26:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac880a5270 msgr2=0x7fac88138ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 --2- 172.21.15.153:0/2754551641 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fac880a5270 0x7fac88138ad0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fac7800d7d0 tx=0x7fac7800dc90 comp rx=0 tx=0).stop 2024-09-10T20:26:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac877fe640 1 -- 172.21.15.153:0/2754551641 reap_dead start 2024-09-10T20:26:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 shutdown_connections 2024-09-10T20:26:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 >> 172.21.15.153:0/2754551641 conn(0x7fac880a96f0 msgr2=0x7fac880a83f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 shutdown_connections 2024-09-10T20:26:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 -- 172.21.15.153:0/2754551641 wait complete. 2024-09-10T20:26:23.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:22.982+0000 7fac8da60640 1 librados: shutdown 2024-09-10T20:26:23.103 INFO:teuthology.orchestra.run.smithi153.stdout:55834574851 2024-09-10T20:26:23.103 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-10T20:26:23.104 INFO:teuthology.orchestra.run.smithi153.stdout:55834574852 2024-09-10T20:26:23.104 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-10T20:26:23.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f1500b02640 10 monclient: get_monmap_and_config 2024-09-10T20:26:23.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f925e16a640 10 monclient: get_monmap_and_config 2024-09-10T20:26:23.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f925e16a640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f925e16a640 10 monclient: monmap: 2024-09-10T20:26:23.250 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.250 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.250 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:23.248704+0000 2024-09-10T20:26:23.250 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:23.248704+0000 2024-09-10T20:26:23.251 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.251 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.251 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:23.251 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:23.251 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:23.251 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:23.251 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:23.251 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f1500b02640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f1500b02640 10 monclient: monmap: 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:23.248694+0000 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:23.248694+0000 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:23.252 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:23.253 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:23.253 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:23.253 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f925e16a640 1 Processor -- start 2024-09-10T20:26:23.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f925e16a640 1 -- start start 2024-09-10T20:26:23.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f925e16a640 10 monclient: init 2024-09-10T20:26:23.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f1500b02640 1 Processor -- start 2024-09-10T20:26:23.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f1500b02640 1 -- start start 2024-09-10T20:26:23.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.242+0000 7f1500b02640 10 monclient: init 2024-09-10T20:26:23.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-10T20:26:23.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f92580a41a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:23.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f92580a4580 0x7f925813d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 10 monclient(hunting): picked mon.noname-c con 0x7f92580a4580 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:23.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a49d0 0x7f925813fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 10 monclient(hunting): picked mon.noname-b con 0x7f92580a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92580a6a20 con 0x7f92580a41a0 2024-09-10T20:26:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f92580a2570 con 0x7f92580a4580 2024-09-10T20:26:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-10T20:26:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f14fc0a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a4580 0x7f14fc13d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 10 monclient(hunting): picked mon.noname-b con 0x7f14fc0a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f14fc0a49a0 0x7f14fc13fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 10 monclient(hunting): picked mon.noname-c con 0x7f14fc0a49a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:23.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14fc0a6a20 con 0x7f14fc0a41a0 2024-09-10T20:26:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f14fc0a2570 con 0x7f14fc0a49a0 2024-09-10T20:26:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f14fc070400 con 0x7f14fc0a4580 2024-09-10T20:26:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9258070400 con 0x7f92580a49d0 2024-09-10T20:26:23.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f1500b02640 10 monclient(hunting): authenticate will time out at 943.388352s 2024-09-10T20:26:23.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f925e16a640 10 monclient(hunting): authenticate will time out at 943.388475s 2024-09-10T20:26:23.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a4580 0x7f14fc13d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a4580 0x7f14fc13d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55610/0 (socket says 172.21.15.153:55610) 2024-09-10T20:26:23.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 1 -- 172.21.15.153:0/3355152412 learned_addr learned my addr 172.21.15.153:0/3355152412 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient(hunting): get_auth_request con 0x7f14fc0a4580 auth_method 0 2024-09-10T20:26:23.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a49d0 0x7f925813fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f92577fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f92580a41a0 0x7f92580a3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60462/0 (socket says 172.21.15.153:60462) 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f92577fe640 1 -- 172.21.15.153:0/2299992445 learned_addr learned my addr 172.21.15.153:0/2299992445 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a49d0 0x7f925813fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55606/0 (socket says 172.21.15.153:55606) 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient(hunting): get_auth_request con 0x7f92580a49d0 auth_method 0 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9256ffd640 1 --2- 172.21.15.153:0/2299992445 >> v2:172.21.15.176:3300/0 conn(0x7f92580a4580 0x7f925813d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9256ffd640 10 monclient(hunting): get_auth_request con 0x7f92580a4580 auth_method 0 2024-09-10T20:26:23.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9256ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9256ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14fad9d640 1 --2- 172.21.15.153:0/3355152412 >> v2:172.21.15.176:3300/0 conn(0x7f14fc0a49a0 0x7f14fc13fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2024-09-10T20:26:23.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 1 -- 172.21.15.153:0/2299992445 >> v2:172.21.15.176:3300/0 conn(0x7f92580a4580 msgr2=0x7f925813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 1 --2- 172.21.15.153:0/2299992445 >> v2:172.21.15.176:3300/0 conn(0x7f92580a4580 0x7f925813d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 1 -- 172.21.15.153:0/2299992445 >> v1:172.21.15.153:6789/0 conn(0x7f92580a41a0 legacy=0x7f92580a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 1 monclient: found mon.noname-b 2024-09-10T20:26:23.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14fad9d640 10 monclient(hunting): get_auth_request con 0x7f14fc0a49a0 auth_method 0 2024-09-10T20:26:23.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f9257fff640 1 -- 172.21.15.153:0/2299992445 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92580a85a0 con 0x7f92580a49d0 2024-09-10T20:26:23.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient(hunting): handle_auth_done global_id 4244 payload 274 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 1 -- 172.21.15.153:0/3355152412 >> v2:172.21.15.176:3300/0 conn(0x7f14fc0a49a0 msgr2=0x7f14fc13fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 1 --2- 172.21.15.153:0/3355152412 >> v2:172.21.15.176:3300/0 conn(0x7f14fc0a49a0 0x7f14fc13fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 1 -- 172.21.15.153:0/3355152412 >> v1:172.21.15.176:6789/0 conn(0x7f14fc0a41a0 legacy=0x7f14fc0a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 1 monclient: found mon.noname-b 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.246+0000 7f14f3fff640 1 -- 172.21.15.153:0/3355152412 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14fc0a85a0 con 0x7f14fc0a4580 2024-09-10T20:26:23.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.250+0000 7f9257fff640 1 -- 172.21.15.153:0/2299992445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a49d0 msgr2=0x7f925813fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.250+0000 7f9257fff640 1 --2- 172.21.15.153:0/2299992445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a49d0 0x7f925813fd10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9250009d90 tx=0x7f925001c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c667c9bc7654ee2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f3fff640 1 -- 172.21.15.153:0/3355152412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a4580 msgr2=0x7f14fc13d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f3fff640 1 --2- 172.21.15.153:0/3355152412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a4580 0x7f14fc13d7c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009870 tx=0x7f14e401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6351e61e1e6f8ccb server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 1 -- 172.21.15.153:0/2299992445 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500ac070 con 0x7f92580a49d0 2024-09-10T20:26:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 10 monclient: dump: 2024-09-10T20:26:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.269 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 1 -- 172.21.15.153:0/2299992445 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92500a68b0 con 0x7f92580a49d0 2024-09-10T20:26:23.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 1 -- 172.21.15.153:0/2299992445 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500a6c80 con 0x7f92580a49d0 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f92567fc640 10 monclient: dump: 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 5 monclient: authenticate success, global_id 4241 2024-09-10T20:26:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 1 -- 172.21.15.153:0/3355152412 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e40b0070 con 0x7f14fc0a4580 2024-09-10T20:26:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:23.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 4 set_mon_vals no callback set 2024-09-10T20:26:23.273 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 10 monclient: dump: 2024-09-10T20:26:23.273 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.273 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.273 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.273 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.273 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 1 -- 172.21.15.153:0/3355152412 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14e40a9710 con 0x7f14fc0a4580 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 1 -- 172.21.15.153:0/3355152412 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14e40a9ae0 con 0x7f14fc0a4580 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 5 monclient: authenticate success, global_id 4244 2024-09-10T20:26:23.274 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 4 set_mon_vals no callback set 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f14f9d9b640 10 monclient: dump: 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.275 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.276 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.276 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 10 monclient: shutdown 2024-09-10T20:26:23.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 -- 172.21.15.153:0/2299992445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a49d0 msgr2=0x7f925813fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 --2- 172.21.15.153:0/2299992445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a49d0 0x7f925813fd10 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9250009d90 tx=0x7f925001c850 comp rx=0 tx=0).stop 2024-09-10T20:26:23.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 -- 172.21.15.153:0/2299992445 shutdown_connections 2024-09-10T20:26:23.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 --2- 172.21.15.153:0/2299992445 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a49d0 0x7f925813fd10 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 --2- 172.21.15.153:0/2299992445 >> v2:172.21.15.176:3300/0 conn(0x7f92580a4580 0x7f925813d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.277 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 -- 172.21.15.153:0/2299992445 >> 172.21.15.153:0/2299992445 conn(0x7f92580ab7e0 msgr2=0x7f92580a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 10 monclient: shutdown 2024-09-10T20:26:23.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 -- 172.21.15.153:0/3355152412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a4580 msgr2=0x7f14fc13d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 --2- 172.21.15.153:0/3355152412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a4580 0x7f14fc13d7c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009870 tx=0x7f14e401c420 comp rx=0 tx=0).stop 2024-09-10T20:26:23.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 -- 172.21.15.153:0/2299992445 shutdown_connections 2024-09-10T20:26:23.278 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 -- 172.21.15.153:0/2299992445 wait complete. 2024-09-10T20:26:23.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 -- 172.21.15.153:0/3355152412 shutdown_connections 2024-09-10T20:26:23.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 --2- 172.21.15.153:0/3355152412 >> v2:172.21.15.176:3300/0 conn(0x7f14fc0a49a0 0x7f14fc13fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 --2- 172.21.15.153:0/3355152412 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a4580 0x7f14fc13d7c0 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009870 tx=0x7f14e401c420 comp rx=0 tx=0).stop 2024-09-10T20:26:23.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 -- 172.21.15.153:0/3355152412 >> 172.21.15.153:0/3355152412 conn(0x7f14fc0ab7e0 msgr2=0x7f14fc0a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 -- 172.21.15.153:0/3355152412 shutdown_connections 2024-09-10T20:26:23.279 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 -- 172.21.15.153:0/3355152412 wait complete. 2024-09-10T20:26:23.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 10 monclient: monmap: 2024-09-10T20:26:23.280 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.280 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:23.280 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:23.280 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:23.280 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.281 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.281 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:23.281 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:23.281 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:23.281 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 librados: starting msgr at 2024-09-10T20:26:23.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 10 monclient: monmap: 2024-09-10T20:26:23.281 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.282 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:23.282 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:23.282 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:23.282 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.282 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.282 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:23.282 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:23.282 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:23.282 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 librados: starting objecter 2024-09-10T20:26:23.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 Processor -- start 2024-09-10T20:26:23.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 -- start start 2024-09-10T20:26:23.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 librados: setting wanted keys 2024-09-10T20:26:23.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 1 librados: calling monclient init 2024-09-10T20:26:23.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f925e16a640 10 monclient: init 2024-09-10T20:26:23.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 librados: starting msgr at 2024-09-10T20:26:23.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.254+0000 7f1500b02640 1 librados: starting objecter 2024-09-10T20:26:23.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 Processor -- start 2024-09-10T20:26:23.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 -- start start 2024-09-10T20:26:23.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 librados: setting wanted keys 2024-09-10T20:26:23.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 librados: calling monclient init 2024-09-10T20:26:23.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient: init 2024-09-10T20:26:23.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:23.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a41a0 0x7f92581370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient(hunting): picked mon.noname-c con 0x7f92580a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92580a4580 0x7f92581375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient(hunting): picked mon.noname-b con 0x7f92580a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92580a49d0 0x7f925813c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient(hunting): picked mon.noname-a con 0x7f92580a49d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92580a49d0 0x7f925813c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92580a49d0 0x7f925813c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42288/0 (socket says 172.21.15.153:42288) 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 1 -- 172.21.15.153:0/2541657017 learned_addr learned my addr 172.21.15.153:0/2541657017 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9256ffd640 1 --2- 172.21.15.153:0/2541657017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92580a4580 0x7f92581375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 1 -- 172.21.15.153:0/2541657017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9258142e00 con 0x7f92580a49d0 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 1 -- 172.21.15.153:0/2541657017 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f9258142c80 con 0x7f92580a4580 2024-09-10T20:26:23.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:26:23.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14fc0a41a0 0x7f14fc073fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient(hunting): picked mon.noname-a con 0x7f14fc0a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:23.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a49a0 0x7f14fc13f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient(hunting): picked mon.noname-c con 0x7f14fc0a49a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14fc13fca0 0x7f14fc1d4710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient(hunting): picked mon.noname-b con 0x7f14fc13fca0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:23.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f14fc142e00 con 0x7f14fc0a41a0 2024-09-10T20:26:23.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f14fc142c80 con 0x7f14fc13fca0 2024-09-10T20:26:23.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f14fc142f80 con 0x7f14fc0a49a0 2024-09-10T20:26:23.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 1 -- 172.21.15.153:0/2541657017 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9258142f80 con 0x7f92580a41a0 2024-09-10T20:26:23.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f1500b02640 10 monclient(hunting): authenticate will time out at 943.400873s 2024-09-10T20:26:23.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a49a0 0x7f14fc13f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14fc13fca0 0x7f14fc1d4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a49a0 0x7f14fc13f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55624/0 (socket says 172.21.15.153:55624) 2024-09-10T20:26:23.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 1 -- 172.21.15.153:0/2085301817 learned_addr learned my addr 172.21.15.153:0/2085301817 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 10 monclient(hunting): get_auth_request con 0x7f14fc0a49a0 auth_method 0 2024-09-10T20:26:23.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient(hunting): get_auth_request con 0x7f14fc13fca0 auth_method 0 2024-09-10T20:26:23.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f925e16a640 10 monclient(hunting): authenticate will time out at 943.401351s 2024-09-10T20:26:23.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient(hunting): get_auth_request con 0x7f92580a49d0 auth_method 0 2024-09-10T20:26:23.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9256ffd640 10 monclient(hunting): get_auth_request con 0x7f92580a4580 auth_method 0 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9256ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9256ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9256ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f92577fe640 1 --2- 172.21.15.153:0/2541657017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a41a0 0x7f92581370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f92577fe640 10 monclient(hunting): get_auth_request con 0x7f92580a41a0 auth_method 0 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f92577fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f92577fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f92577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fa59c640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14fc0a41a0 0x7f14fc073fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fa59c640 10 monclient(hunting): get_auth_request con 0x7f14fc0a41a0 auth_method 0 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fa59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fa59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fa59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fa59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fa59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fa59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient(hunting): handle_auth_done global_id 4240 payload 824 2024-09-10T20:26:23.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f92577fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f92577fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f92577fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9256ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 1 -- 172.21.15.153:0/2085301817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a49a0 msgr2=0x7f14fc13f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a49a0 0x7f14fc13f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 1 -- 172.21.15.153:0/2085301817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14fc0a41a0 msgr2=0x7f14fc073fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14fc0a41a0 0x7f14fc073fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 1 monclient: found mon.noname-b 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14fad9d640 1 -- 172.21.15.153:0/2085301817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14fc12d990 con 0x7f14fc13fca0 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f14f3fff640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a49a0 0x7f14fc13f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient(hunting): handle_auth_done global_id 4254 payload 824 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 1 -- 172.21.15.153:0/2541657017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a41a0 msgr2=0x7f92581370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 1 --2- 172.21.15.153:0/2541657017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a41a0 0x7f92581370a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 1 -- 172.21.15.153:0/2541657017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92580a4580 msgr2=0x7f92581375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 1 --2- 172.21.15.153:0/2541657017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92580a4580 0x7f92581375e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 1 monclient: found mon.noname-a 2024-09-10T20:26:23.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.258+0000 7f9257fff640 1 -- 172.21.15.153:0/2541657017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9258121f60 con 0x7f92580a49d0 2024-09-10T20:26:23.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14fad9d640 1 -- 172.21.15.153:0/2085301817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14fc13fca0 msgr2=0x7f14fc1d4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14fad9d640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14fc13fca0 0x7f14fc1d4710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14f400efc0 tx=0x7f14f400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 client.?.objecter ms_handle_connect 0x7f14fc13fca0 2024-09-10T20:26:23.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:23.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f9257fff640 1 -- 172.21.15.153:0/2541657017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92580a49d0 msgr2=0x7f925813c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f9257fff640 1 --2- 172.21.15.153:0/2541657017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92580a49d0 0x7f925813c9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f925001c680 tx=0x7f92500195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 client.?.objecter ms_handle_connect 0x7f92580a49d0 2024-09-10T20:26:23.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:23.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 1 -- 172.21.15.153:0/2085301817 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f4018020 con 0x7f14fc13fca0 2024-09-10T20:26:23.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:23.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 monclient: dump: 2024-09-10T20:26:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.301 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 1 -- 172.21.15.153:0/2085301817 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14f40146a0 con 0x7f14fc13fca0 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 5 monclient: authenticate success, global_id 4240 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 10 monclient: _renew_subs 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14fc0a21c0 con 0x7f14fc13fca0 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f959a640 4 set_mon_vals no callback set 2024-09-10T20:26:23.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 10 client.4240.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:23.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 1 -- 172.21.15.153:0/2541657017 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500c0070 con 0x7f92580a49d0 2024-09-10T20:26:23.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:23.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 monclient: dump: 2024-09-10T20:26:23.303 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.303 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.304 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.304 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.304 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.304 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.304 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.304 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.304 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 1 -- 172.21.15.153:0/2541657017 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92500a7e50 con 0x7f92580a49d0 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f925e16a640 5 monclient: authenticate success, global_id 4254 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f925e16a640 10 monclient: _renew_subs 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f925e16a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f925e16a640 1 -- 172.21.15.153:0/2541657017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92580aa1d0 con 0x7f92580a49d0 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 1 -- 172.21.15.153:0/2541657017 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92500ac050 con 0x7f92580a49d0 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 monclient: dump: 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.306 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f925e16a640 10 client.4254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f925e16a640 10 monclient: _renew_subs 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f925e16a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f925e16a640 1 -- 172.21.15.153:0/2541657017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9258121c80 con 0x7f92580a49d0 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f925e16a640 1 librados: init done 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f9255ffb640 4 set_mon_vals no callback set 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 1 -- 172.21.15.153:0/2541657017 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f92500af070 con 0x7f92580a49d0 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 1 --2- 172.21.15.153:0/2541657017 >> v2:172.21.15.176:6812/3003473398 conn(0x7f924403cef0 0x7f924403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 1 -- 172.21.15.153:0/2541657017 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f92500e9b70 con 0x7f92580a49d0 2024-09-10T20:26:23.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 10 client.4254.objecter ms_dispatch 0x7f9258000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 3 client.4254.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 3 client.4254.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923ffff640 20 client.4254.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 10 monclient: _renew_subs 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14fc12e230 con 0x7f14fc13fca0 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 1 librados: init done 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14fc0a3720 con 0x7f14fc13fca0 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14fad9d640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14fc13fca0 0x7f14fc1d4710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14f400efc0 tx=0x7f14f400c490 comp rx=0 tx=0).queue_received will delay after 1.726e+09 on 0x7f14f404fb60 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 1 -- 172.21.15.153:0/2085301817 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14f4014a90 con 0x7f14fc13fca0 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 10 monclient: dump: 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.310 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 1 -- 172.21.15.153:0/2085301817 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f14f401b020 con 0x7f14fc13fca0 2024-09-10T20:26:23.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923dffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923dffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923dffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f923dffb640 1 -- 172.21.15.153:0/2541657017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925813f370 con 0x7f92580a49d0 2024-09-10T20:26:23.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f92577fe640 1 --2- 172.21.15.153:0/2541657017 >> v2:172.21.15.176:6812/3003473398 conn(0x7f924403cef0 0x7f924403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.262+0000 7f14f2ffd640 1 --2- 172.21.15.153:0/2085301817 >> v2:172.21.15.176:6812/3003473398 conn(0x7f14dc03ce50 0x7f14dc03f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.266+0000 7f14fa59c640 1 --2- 172.21.15.153:0/2085301817 >> v2:172.21.15.176:6812/3003473398 conn(0x7f14dc03ce50 0x7f14dc03f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.266+0000 7f14fa59c640 10 monclient: get_auth_request con 0x7f14dc03ce50 auth_method 0 2024-09-10T20:26:23.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.266+0000 7f14fa59c640 1 --2- 172.21.15.153:0/2085301817 >> v2:172.21.15.176:6812/3003473398 conn(0x7f14dc03ce50 0x7f14dc03f310 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f14ec007fd0 tx=0x7f14ec007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.266+0000 7f14f2ffd640 10 client.4240.objecter ms_handle_connect 0x7f14dc03ce50 2024-09-10T20:26:23.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.270+0000 7f92577fe640 10 monclient: get_auth_request con 0x7f924403cef0 auth_method 0 2024-09-10T20:26:23.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.270+0000 7f923ffff640 1 -- 172.21.15.153:0/2541657017 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f9250016560 con 0x7f92580a49d0 2024-09-10T20:26:23.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.270+0000 7f923ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.270+0000 7f923ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:23.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.270+0000 7f92577fe640 1 --2- 172.21.15.153:0/2541657017 >> v2:172.21.15.176:6812/3003473398 conn(0x7f924403cef0 0x7f924403f3b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9248007ca0 tx=0x7f9248007bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.270+0000 7f923ffff640 10 client.4254.objecter ms_handle_connect 0x7f924403cef0 2024-09-10T20:26:23.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.278+0000 7f14f2ffd640 1 -- 172.21.15.153:0/2085301817 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f14f404fb60 con 0x7f14fc13fca0 2024-09-10T20:26:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.278+0000 7f14f2ffd640 10 client.4240.objecter ms_dispatch 0x7f14fc000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.278+0000 7f14f2ffd640 3 client.4240.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.278+0000 7f14f2ffd640 3 client.4240.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.278+0000 7f14f2ffd640 20 client.4240.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.281+0000 7f31b549b640 10 monclient: get_monmap_and_config 2024-09-10T20:26:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.281+0000 7f31b549b640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.281+0000 7f31b549b640 10 monclient: monmap: 2024-09-10T20:26:23.316 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.316 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.316 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:23.289583+0000 2024-09-10T20:26:23.316 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:23.289583+0000 2024-09-10T20:26:23.316 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.316 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.316 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:23.316 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:23.316 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:23.317 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:23.317 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:23.317 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient: get_monmap_and_config 2024-09-10T20:26:23.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient: monmap: 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:23.289935+0000 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:23.289935+0000 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:23.318 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:23.319 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 1 Processor -- start 2024-09-10T20:26:23.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 1 -- start start 2024-09-10T20:26:23.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient: init 2024-09-10T20:26:23.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 1 Processor -- start 2024-09-10T20:26:23.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 1 -- start start 2024-09-10T20:26:23.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient: init 2024-09-10T20:26:23.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T20:26:23.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a41a0 0x7f31b00a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient(hunting): picked mon.noname-b con 0x7f31b00a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a41a0 0x7f31b00a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a41a0 0x7f31b00a3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55648/0 (socket says 172.21.15.153:55648) 2024-09-10T20:26:23.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-10T20:26:23.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 1 -- 172.21.15.153:0/1476539878 learned_addr learned my addr 172.21.15.153:0/1476539878 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31b00a4580 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:23.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 1 --2- 172.21.15.153:0/1476539878 >> v2:172.21.15.176:3300/0 conn(0x7f31b00a49a0 0x7f31b00a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25380a37d0 0x7f25380a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient(hunting): picked mon.noname-b con 0x7f25380a37d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f253809cfb0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:23.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient(hunting): picked mon.noname-c con 0x7f31b00a49a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:23.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 1 -- 172.21.15.153:0/1476539878 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f31b00a6a20 con 0x7f31b00a4580 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 1 -- 172.21.15.153:0/1476539878 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f31b00a2570 con 0x7f31b00a49a0 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 1 -- 172.21.15.153:0/1476539878 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f31b0070400 con 0x7f31b00a41a0 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31b549b640 10 monclient(hunting): authenticate will time out at 943.429335s 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient(hunting): get_auth_request con 0x7f31b00a41a0 auth_method 0 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253da28640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f253809fa10 0x7f2538135d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60486/0 (socket says 172.21.15.153:60486) 2024-09-10T20:26:23.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253da28640 1 -- 172.21.15.153:0/3991669694 learned_addr learned my addr 172.21.15.153:0/3991669694 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:23.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f253809fa10 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:23.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 1 -- 172.21.15.153:0/3991669694 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2538136480 con 0x7f253809fa10 2024-09-10T20:26:23.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 1 -- 172.21.15.153:0/3991669694 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f253809d430 con 0x7f253809cfb0 2024-09-10T20:26:23.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 1 -- 172.21.15.153:0/3991669694 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f253809eb20 con 0x7f25380a37d0 2024-09-10T20:26:23.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253f48b640 10 monclient(hunting): authenticate will time out at 943.429579s 2024-09-10T20:26:23.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 1 --2- 172.21.15.153:0/3991669694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25380a37d0 0x7f25380a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient(hunting): get_auth_request con 0x7f25380a37d0 auth_method 0 2024-09-10T20:26:23.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31af7fe640 1 --2- 172.21.15.153:0/1476539878 >> v2:172.21.15.176:3300/0 conn(0x7f31b00a49a0 0x7f31b00a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31af7fe640 10 monclient(hunting): get_auth_request con 0x7f31b00a49a0 auth_method 0 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient(hunting): handle_auth_done global_id 4253 payload 274 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 1 -- 172.21.15.153:0/1476539878 >> v2:172.21.15.176:3300/0 conn(0x7f31b00a49a0 msgr2=0x7f31b00a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 1 --2- 172.21.15.153:0/1476539878 >> v2:172.21.15.176:3300/0 conn(0x7f31b00a49a0 0x7f31b00a60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 1 -- 172.21.15.153:0/1476539878 >> v1:172.21.15.153:6789/0 conn(0x7f31b00a4580 legacy=0x7f31b00a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 1 monclient: found mon.noname-b 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f31aeffd640 1 -- 172.21.15.153:0/1476539878 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b00a3c80 con 0x7f31b00a41a0 2024-09-10T20:26:23.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient(hunting): handle_auth_done global_id 4256 payload 274 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 1 -- 172.21.15.153:0/3991669694 >> v1:172.21.15.176:6789/0 conn(0x7f253809cfb0 legacy=0x7f2538135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 1 -- 172.21.15.153:0/3991669694 >> v1:172.21.15.153:6789/0 conn(0x7f253809fa10 legacy=0x7f2538135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 1 monclient: found mon.noname-b 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.285+0000 7f253d227640 1 -- 172.21.15.153:0/3991669694 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25380a5230 con 0x7f25380a37d0 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.289+0000 7f31aeffd640 1 -- 172.21.15.153:0/1476539878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a41a0 msgr2=0x7f31b00a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.289+0000 7f31aeffd640 1 --2- 172.21.15.153:0/1476539878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a41a0 0x7f31b00a3740 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009870 tx=0x7f31a001c440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bdf2d7b42cbc3a1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 1 -- 172.21.15.153:0/3991669694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25380a37d0 msgr2=0x7f25380a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 1 --2- 172.21.15.153:0/3991669694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25380a37d0 0x7f25380a3bb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25340097f0 tx=0x7f253401c430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2869650f1d2e9016 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 1 -- 172.21.15.153:0/1476539878 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a00b0070 con 0x7f31b00a41a0 2024-09-10T20:26:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 10 monclient: dump: 2024-09-10T20:26:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.332 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 1 -- 172.21.15.153:0/1476539878 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31a00ab600 con 0x7f31b00a41a0 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 1 -- 172.21.15.153:0/1476539878 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a00ab9d0 con 0x7f31b00a41a0 2024-09-10T20:26:23.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31adffb640 10 monclient: dump: 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.334 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 5 monclient: authenticate success, global_id 4253 2024-09-10T20:26:23.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:23.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 4 set_mon_vals no callback set 2024-09-10T20:26:23.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 1 -- 172.21.15.153:0/3991669694 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25340b0070 con 0x7f25380a37d0 2024-09-10T20:26:23.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 10 monclient: dump: 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.336 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 1 -- 172.21.15.153:0/3991669694 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25340a97f0 con 0x7f25380a37d0 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 1 -- 172.21.15.153:0/3991669694 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25340a9bc0 con 0x7f25380a37d0 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f252ffff640 10 monclient: dump: 2024-09-10T20:26:23.337 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 5 monclient: authenticate success, global_id 4256 2024-09-10T20:26:23.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:23.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 4 set_mon_vals no callback set 2024-09-10T20:26:23.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient: shutdown 2024-09-10T20:26:23.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 -- 172.21.15.153:0/3991669694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25380a37d0 msgr2=0x7f25380a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 --2- 172.21.15.153:0/3991669694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25380a37d0 0x7f25380a3bb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25340097f0 tx=0x7f253401c430 comp rx=0 tx=0).stop 2024-09-10T20:26:23.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 10 monclient: shutdown 2024-09-10T20:26:23.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 -- 172.21.15.153:0/1476539878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a41a0 msgr2=0x7f31b00a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 --2- 172.21.15.153:0/1476539878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a41a0 0x7f31b00a3740 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009870 tx=0x7f31a001c440 comp rx=0 tx=0).stop 2024-09-10T20:26:23.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 -- 172.21.15.153:0/3991669694 shutdown_connections 2024-09-10T20:26:23.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 --2- 172.21.15.153:0/3991669694 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f25380a37d0 0x7f25380a3bb0 secure :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f25340097f0 tx=0x7f253401c430 comp rx=0 tx=0).stop 2024-09-10T20:26:23.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 -- 172.21.15.153:0/3991669694 >> 172.21.15.153:0/3991669694 conn(0x7f25380a8480 msgr2=0x7f25380a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 -- 172.21.15.153:0/1476539878 shutdown_connections 2024-09-10T20:26:23.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 --2- 172.21.15.153:0/1476539878 >> v2:172.21.15.176:3300/0 conn(0x7f31b00a49a0 0x7f31b00a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 --2- 172.21.15.153:0/1476539878 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a41a0 0x7f31b00a3740 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 -- 172.21.15.153:0/1476539878 >> 172.21.15.153:0/1476539878 conn(0x7f31b00ab7e0 msgr2=0x7f31b00a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 -- 172.21.15.153:0/3991669694 shutdown_connections 2024-09-10T20:26:23.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 -- 172.21.15.153:0/1476539878 shutdown_connections 2024-09-10T20:26:23.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 -- 172.21.15.153:0/1476539878 wait complete. 2024-09-10T20:26:23.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 -- 172.21.15.153:0/3991669694 wait complete. 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient: monmap: 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 librados: starting msgr at 2024-09-10T20:26:23.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 librados: starting objecter 2024-09-10T20:26:23.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 Processor -- start 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 10 monclient: monmap: 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:23.345 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:23.346 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 librados: starting msgr at 2024-09-10T20:26:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 librados: starting objecter 2024-09-10T20:26:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 Processor -- start 2024-09-10T20:26:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f31b549b640 1 -- start start 2024-09-10T20:26:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 1 librados: setting wanted keys 2024-09-10T20:26:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 1 librados: calling monclient init 2024-09-10T20:26:23.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient: init 2024-09-10T20:26:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 -- start start 2024-09-10T20:26:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 librados: setting wanted keys 2024-09-10T20:26:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 librados: calling monclient init 2024-09-10T20:26:23.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient: init 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f253809cfb0 0x7f25381ddf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient(hunting): picked mon.noname-c con 0x7f253809cfb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253809fa10 0x7f25381de4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient(hunting): picked mon.noname-b con 0x7f253809fa10 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25381322c0 0x7f2538132750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient(hunting): picked mon.noname-a con 0x7f25381322c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2538141780 con 0x7f25381322c0 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2538141900 con 0x7f253809fa10 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2538141a80 con 0x7f253809cfb0 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253f48b640 10 monclient(hunting): authenticate will time out at 943.439462s 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f253809cfb0 0x7f25381ddf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253da28640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25381322c0 0x7f2538132750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f253809cfb0 0x7f25381ddf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55662/0 (socket says 172.21.15.153:55662) 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 1 -- 172.21.15.153:0/1974313033 learned_addr learned my addr 172.21.15.153:0/1974313033 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 10 monclient(hunting): get_auth_request con 0x7f253809cfb0 auth_method 0 2024-09-10T20:26:23.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253da28640 10 monclient(hunting): get_auth_request con 0x7f25381322c0 auth_method 0 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253da28640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253da28640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253da28640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253da28640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253da28640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253da28640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253d227640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253ca26640 1 --2- 172.21.15.153:0/1974313033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253809fa10 0x7f25381de4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253ca26640 10 monclient(hunting): get_auth_request con 0x7f253809fa10 auth_method 0 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253ca26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253ca26640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.293+0000 7f253ca26640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 10 monclient(hunting): handle_auth_done global_id 4260 payload 824 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 1 -- 172.21.15.153:0/1974313033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f253809cfb0 msgr2=0x7f25381ddf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 1 --2- 172.21.15.153:0/1974313033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f253809cfb0 0x7f25381ddf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 1 -- 172.21.15.153:0/1974313033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253809fa10 msgr2=0x7f25381de4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 1 --2- 172.21.15.153:0/1974313033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253809fa10 0x7f25381de4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 1 monclient: found mon.noname-a 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 1 -- 172.21.15.153:0/1974313033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2538136a40 con 0x7f25381322c0 2024-09-10T20:26:23.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253ca26640 1 --2- 172.21.15.153:0/1974313033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253809fa10 0x7f25381de4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:26:23.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:26:23.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31b00a41a0 0x7f31b0123940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient(hunting): picked mon.noname-b con 0x7f31b00a41a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:23.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b00a4580 0x7f31b0125960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient: get_monmap_and_config 2024-09-10T20:26:23.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient: monmap: 2024-09-10T20:26:23.353 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.353 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.353 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:23.302950+0000 2024-09-10T20:26:23.353 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:23.302950+0000 2024-09-10T20:26:23.353 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.353 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.354 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:23.354 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:23.354 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:23.354 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:23.354 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:23.354 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient(hunting): picked mon.noname-a con 0x7f31b00a4580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a49a0 0x7f31b013c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient(hunting): picked mon.noname-c con 0x7f31b00a49a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f31b0143210 con 0x7f31b00a4580 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b00a4580 0x7f31b0125960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f31b00a6830 con 0x7f31b00a41a0 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b00a4580 0x7f31b0125960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42298/0 (socket says 172.21.15.153:42298) 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 1 -- 172.21.15.153:0/1442100865 learned_addr learned my addr 172.21.15.153:0/1442100865 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31af7fe640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a49a0 0x7f31b013c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f31b0143390 con 0x7f31b00a49a0 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31aeffd640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31b00a41a0 0x7f31b0123940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31b549b640 10 monclient(hunting): authenticate will time out at 943.441465s 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient(hunting): get_auth_request con 0x7f31b00a4580 auth_method 0 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31af7fe640 10 monclient(hunting): get_auth_request con 0x7f31b00a49a0 auth_method 0 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31aeffd640 10 monclient(hunting): get_auth_request con 0x7f31b00a41a0 auth_method 0 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 1 Processor -- start 2024-09-10T20:26:23.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2024-09-10T20:26:23.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 1 -- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a49a0 msgr2=0x7f31b013c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a49a0 0x7f31b013c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 1 -- start start 2024-09-10T20:26:23.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient: init 2024-09-10T20:26:23.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 1 -- 172.21.15.153:0/1442100865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31b00a41a0 msgr2=0x7f31b0123940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31b00a41a0 0x7f31b0123940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 1 monclient: found mon.noname-a 2024-09-10T20:26:23.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31ae7fc640 1 -- 172.21.15.153:0/1442100865 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b013d470 con 0x7f31b00a4580 2024-09-10T20:26:23.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f31af7fe640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a49a0 0x7f31b013c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:23.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient: _add_conns ranks=[4,1,2,3,0] 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f77a80a37d0 0x7f77a80a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient(hunting): picked mon.noname-c con 0x7f77a80a37d0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f77a809cfb0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a80a40f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient(hunting): picked mon.noname-b con 0x7f77a809fa10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77a809d430 con 0x7f77a809cfb0 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a80a40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a80a40f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55690/0 (socket says 172.21.15.153:55690) 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 1 -- 172.21.15.153:0/4253992325 learned_addr learned my addr 172.21.15.153:0/4253992325 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 1 -- 172.21.15.153:0/4253992325 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f77a809eb20 con 0x7f77a80a37d0 2024-09-10T20:26:23.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 1 -- 172.21.15.153:0/4253992325 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f77a809ed70 con 0x7f77a809fa10 2024-09-10T20:26:23.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77a7fff640 1 --2- 172.21.15.153:0/4253992325 >> v2:172.21.15.176:3300/0 conn(0x7f77a80a37d0 0x7f77a80a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77ae502640 10 monclient(hunting): authenticate will time out at 943.442720s 2024-09-10T20:26:23.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 1 -- 172.21.15.153:0/1974313033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25381322c0 msgr2=0x7f2538132750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253da28640 1 --2- 172.21.15.153:0/1974313033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25381322c0 0x7f2538132750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f253000caa0 tx=0x7f253000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 client.?.objecter ms_handle_connect 0x7f25381322c0 2024-09-10T20:26:23.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:23.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient(hunting): get_auth_request con 0x7f77a809fa10 auth_method 0 2024-09-10T20:26:23.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77a7fff640 10 monclient(hunting): get_auth_request con 0x7f77a80a37d0 auth_method 0 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77a7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77a7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77a7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2024-09-10T20:26:23.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 1 -- 172.21.15.153:0/4253992325 >> v2:172.21.15.176:3300/0 conn(0x7f77a80a37d0 msgr2=0x7f77a80a3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 1 --2- 172.21.15.153:0/4253992325 >> v2:172.21.15.176:3300/0 conn(0x7f77a80a37d0 0x7f77a80a3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 1 -- 172.21.15.153:0/4253992325 >> v1:172.21.15.176:6789/0 conn(0x7f77a809cfb0 legacy=0x7f77a8135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 1 monclient: found mon.noname-b 2024-09-10T20:26:23.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f77aca9f640 1 -- 172.21.15.153:0/4253992325 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a80a5230 con 0x7f77a809fa10 2024-09-10T20:26:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 1 -- 172.21.15.153:0/1974313033 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f253000ac10 con 0x7f25381322c0 2024-09-10T20:26:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 monclient: dump: 2024-09-10T20:26:23.368 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 1 -- 172.21.15.153:0/1974313033 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f253000add0 con 0x7f25381322c0 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252f7fe640 4 set_mon_vals no callback set 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253f48b640 5 monclient: authenticate success, global_id 4260 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 1 -- 172.21.15.153:0/1974313033 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2530034650 con 0x7f25381322c0 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253f48b640 10 monclient: _renew_subs 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253f48b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253f48b640 1 -- 172.21.15.153:0/1974313033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f253809f0b0 con 0x7f25381322c0 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f252dffb640 10 monclient: dump: 2024-09-10T20:26:23.370 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253f48b640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253f48b640 10 monclient: _renew_subs 2024-09-10T20:26:23.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253f48b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253f48b640 1 -- 172.21.15.153:0/1974313033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2538137ca0 con 0x7f25381322c0 2024-09-10T20:26:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.297+0000 7f253f48b640 1 librados: init done 2024-09-10T20:26:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient: get_monmap_and_config 2024-09-10T20:26:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient: monmap: 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:23.305923+0000 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:23.305923+0000 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:23.373 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:23.374 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31ae7fc640 1 -- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b00a4580 msgr2=0x7f31b0125960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31ae7fc640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b00a4580 0x7f31b0125960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c450 tx=0x7f31a400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 1 Processor -- start 2024-09-10T20:26:23.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f251b7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f251b7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f251b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 1 -- start start 2024-09-10T20:26:23.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient: init 2024-09-10T20:26:23.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f251b7fe640 1 -- 172.21.15.153:0/1974313033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25380a3bb0 con 0x7f25381322c0 2024-09-10T20:26:23.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 client.?.objecter ms_handle_connect 0x7f31b00a4580 2024-09-10T20:26:23.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:23.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 1 -- 172.21.15.153:0/1442100865 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4012a40 con 0x7f31b00a4580 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 monclient: dump: 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.377 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.378 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 1 -- 172.21.15.153:0/1442100865 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31a4017070 con 0x7f31b00a4580 2024-09-10T20:26:23.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 1 -- 172.21.15.153:0/1442100865 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4012e40 con 0x7f31b00a4580 2024-09-10T20:26:23.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31977fe640 10 monclient: dump: 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.379 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient: _add_conns ranks=[2,3,4,1,0] 2024-09-10T20:26:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 0x7fe45809bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient(hunting): picked mon.noname-b con 0x7fe4580a2240 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe4580a2620 0x7fe4580a2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient(hunting): picked mon.noname-a con 0x7fe4580a2620 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe4581366a0 0x7fe458136af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient(hunting): picked mon.noname-c con 0x7fe4581366a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:23.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fe45809b8e0 con 0x7fe4580a2620 2024-09-10T20:26:23.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe4580a3740 con 0x7fe4581366a0 2024-09-10T20:26:23.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe4580a1730 con 0x7fe4580a2240 2024-09-10T20:26:23.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe45c92c640 10 monclient(hunting): authenticate will time out at 943.446027s 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 5 monclient: authenticate success, global_id 4263 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31ad7fa640 4 set_mon_vals no callback set 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 10 monclient: _renew_subs 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31b00a21c0 con 0x7f31b00a4580 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 10 monclient: _renew_subs 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31b0127160 con 0x7f31b00a4580 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 1 librados: init done 2024-09-10T20:26:23.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f31b00a3740 con 0x7f31b00a4580 2024-09-10T20:26:23.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe4581366a0 0x7fe458136af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe4581366a0 0x7fe458136af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.153:40000/0 (socket says 172.21.15.153:40000) 2024-09-10T20:26:23.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 1 -- 172.21.15.153:0/127339032 learned_addr learned my addr 172.21.15.153:0/127339032 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient(hunting): get_auth_request con 0x7fe4581366a0 auth_method 0 2024-09-10T20:26:23.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f252dffb640 1 -- 172.21.15.153:0/1974313033 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f25300347f0 con 0x7f25381322c0 2024-09-10T20:26:23.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f252dffb640 1 --2- 172.21.15.153:0/1974313033 >> v2:172.21.15.176:6812/3003473398 conn(0x7f251c03cef0 0x7f251c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f252dffb640 1 -- 172.21.15.153:0/1974313033 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f25300721e0 con 0x7f25381322c0 2024-09-10T20:26:23.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f252dffb640 10 client.4260.objecter ms_dispatch 0x7f2538000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f252dffb640 3 client.4260.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f252dffb640 3 client.4260.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f252dffb640 20 client.4260.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient(hunting): handle_auth_done global_id 4258 payload 274 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 1 -- 172.21.15.153:0/127339032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 msgr2=0x7fe45809bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 1 --2- 172.21.15.153:0/127339032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 0x7fe45809bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 1 -- 172.21.15.153:0/127339032 >> v2:172.21.15.153:3300/0 conn(0x7fe4580a2620 msgr2=0x7fe4580a2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 1 --2- 172.21.15.153:0/127339032 >> v2:172.21.15.153:3300/0 conn(0x7fe4580a2620 0x7fe4580a2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 1 monclient: found mon.noname-c 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7fe456d9d640 1 -- 172.21.15.153:0/127339032 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe45809b600 con 0x7fe4581366a0 2024-09-10T20:26:23.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f253d227640 1 --2- 172.21.15.153:0/1974313033 >> v2:172.21.15.176:6812/3003473398 conn(0x7f251c03cef0 0x7f251c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.301+0000 7f253d227640 10 monclient: get_auth_request con 0x7f251c03cef0 auth_method 0 2024-09-10T20:26:23.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77aca9f640 1 -- 172.21.15.153:0/4253992325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 msgr2=0x7f77a80a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77aca9f640 1 --2- 172.21.15.153:0/4253992325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a80a40f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009e50 tx=0x7f77a001c980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a162e644428ac8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 1 -- 172.21.15.153:0/4253992325 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00a57f0 con 0x7f77a809fa10 2024-09-10T20:26:23.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:23.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 10 monclient: dump: 2024-09-10T20:26:23.387 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.387 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 1 -- 172.21.15.153:0/4253992325 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77a00a5990 con 0x7f77a809fa10 2024-09-10T20:26:23.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 1 -- 172.21.15.153:0/4253992325 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77a00a5d60 con 0x7f77a809fa10 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a77fe640 10 monclient: dump: 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.389 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 5 monclient: authenticate success, global_id 4265 2024-09-10T20:26:23.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:23.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 4 set_mon_vals no callback set 2024-09-10T20:26:23.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient: shutdown 2024-09-10T20:26:23.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 -- 172.21.15.153:0/4253992325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 msgr2=0x7f77a80a40f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 --2- 172.21.15.153:0/4253992325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a80a40f0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f77a0009e50 tx=0x7f77a001c980 comp rx=0 tx=0).stop 2024-09-10T20:26:23.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f253d227640 1 --2- 172.21.15.153:0/1974313033 >> v2:172.21.15.176:6812/3003473398 conn(0x7f251c03cef0 0x7f251c03f3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f25340097f0 tx=0x7f2534008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f252dffb640 10 client.4260.objecter ms_handle_connect 0x7f251c03cef0 2024-09-10T20:26:23.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 -- 172.21.15.153:0/4253992325 shutdown_connections 2024-09-10T20:26:23.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 --2- 172.21.15.153:0/4253992325 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a80a40f0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 --2- 172.21.15.153:0/4253992325 >> v2:172.21.15.176:3300/0 conn(0x7f77a80a37d0 0x7f77a80a3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 -- 172.21.15.153:0/4253992325 >> 172.21.15.153:0/4253992325 conn(0x7f77a80a8480 msgr2=0x7f77a80a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 -- 172.21.15.153:0/4253992325 shutdown_connections 2024-09-10T20:26:23.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 -- 172.21.15.153:0/4253992325 wait complete. 2024-09-10T20:26:23.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient: monmap: 2024-09-10T20:26:23.391 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 librados: starting msgr at 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 librados: starting objecter 2024-09-10T20:26:23.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 Processor -- start 2024-09-10T20:26:23.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe456d9d640 1 -- 172.21.15.153:0/127339032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4581366a0 msgr2=0x7fe458136af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe456d9d640 1 --2- 172.21.15.153:0/127339032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4581366a0 0x7fe458136af0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe4580a52f0 tx=0x7fe45001c870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fe1d69c001a1a84b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 1 -- 172.21.15.153:0/127339032 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4500a56f0 con 0x7fe4581366a0 2024-09-10T20:26:23.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:23.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 10 monclient: dump: 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 1 -- 172.21.15.153:0/127339032 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4500a5890 con 0x7fe4581366a0 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 1 -- 172.21.15.153:0/127339032 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4500a5c60 con 0x7fe4581366a0 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45559a640 10 monclient: dump: 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.395 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.396 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.396 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.396 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.396 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.396 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45c92c640 5 monclient: authenticate success, global_id 4258 2024-09-10T20:26:23.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45c92c640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:23.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7fe45c92c640 4 set_mon_vals no callback set 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 -- start start 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 librados: setting wanted keys 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 librados: calling monclient init 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient: init 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 0x7f77a81dda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient(hunting): picked mon.noname-a con 0x7f77a809cfb0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a81ddf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient(hunting): picked mon.noname-c con 0x7f77a809fa10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77a80a37d0 0x7f77a8133360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient(hunting): picked mon.noname-b con 0x7f77a80a37d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f77a80775c0 con 0x7f77a809cfb0 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f77a8077440 con 0x7f77a80a37d0 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f77a8077740 con 0x7f77a809fa10 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77ae502640 10 monclient(hunting): authenticate will time out at 943.451596s 2024-09-10T20:26:23.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 0x7f77a81dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 0x7f77a81dda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42320/0 (socket says 172.21.15.153:42320) 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 1 -- 172.21.15.153:0/552200403 learned_addr learned my addr 172.21.15.153:0/552200403 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 10 monclient(hunting): get_auth_request con 0x7f77a809cfb0 auth_method 0 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f779ffff640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a81ddf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f779ffff640 10 monclient(hunting): get_auth_request con 0x7f77a809fa10 auth_method 0 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f779ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f779ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f779ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 1 -- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 msgr2=0x7f77a81dda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77aca9f640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77a80a37d0 0x7f77a8133360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 1 -- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 msgr2=0x7f77a81dda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.305+0000 7f77a7fff640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 0x7f77a81dda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a7fff640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 0x7f77a81dda40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77aca9f640 10 monclient(hunting): get_auth_request con 0x7f77a80a37d0 auth_method 0 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77aca9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77aca9f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77aca9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77aca9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77aca9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77aca9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 10 monclient(hunting): handle_auth_done global_id 4268 payload 824 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 1 -- 172.21.15.153:0/552200403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77a80a37d0 msgr2=0x7f77a8133360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77a80a37d0 0x7f77a8133360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 1 -- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 msgr2=0x7f77a81dda40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 0x7f77a81dda40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 1 monclient: found mon.noname-c 2024-09-10T20:26:23.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 1 -- 172.21.15.153:0/552200403 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77a80a72a0 con 0x7f77a809fa10 2024-09-10T20:26:23.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f252dffb640 1 -- 172.21.15.153:0/1974313033 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f2538136a40 con 0x7f25381322c0 2024-09-10T20:26:23.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f252dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f252dffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 10 monclient: shutdown 2024-09-10T20:26:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 -- 172.21.15.153:0/127339032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4581366a0 msgr2=0x7fe458136af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 --2- 172.21.15.153:0/127339032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4581366a0 0x7fe458136af0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fe4580a52f0 tx=0x7fe45001c870 comp rx=0 tx=0).stop 2024-09-10T20:26:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 -- 172.21.15.153:0/127339032 shutdown_connections 2024-09-10T20:26:23.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 --2- 172.21.15.153:0/127339032 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4581366a0 0x7fe458136af0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 --2- 172.21.15.153:0/127339032 >> v2:172.21.15.153:3300/0 conn(0x7fe4580a2620 0x7fe4580a2a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 --2- 172.21.15.153:0/127339032 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 0x7fe45809bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 -- 172.21.15.153:0/127339032 >> 172.21.15.153:0/127339032 conn(0x7fe4580a7830 msgr2=0x7fe4580a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 -- 172.21.15.153:0/127339032 shutdown_connections 2024-09-10T20:26:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 -- 172.21.15.153:0/127339032 wait complete. 2024-09-10T20:26:23.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f31977fe640 1 -- 172.21.15.153:0/1442100865 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f31a401c460 con 0x7f31b00a4580 2024-09-10T20:26:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 10 monclient: build_initial_monmap 2024-09-10T20:26:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f31977fe640 1 --2- 172.21.15.153:0/1442100865 >> v2:172.21.15.176:6812/3003473398 conn(0x7f319c03cf20 0x7f319c03f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 10 monclient: monmap: 2024-09-10T20:26:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:23.406 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:23.407 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:23.407 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f31977fe640 1 -- 172.21.15.153:0/1442100865 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f31a40249e0 con 0x7f31b00a4580 2024-09-10T20:26:23.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f31977fe640 10 client.4263.objecter ms_dispatch 0x7f31b0000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f31977fe640 3 client.4263.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f31977fe640 3 client.4263.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f31977fe640 20 client.4263.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f31aeffd640 1 --2- 172.21.15.153:0/1442100865 >> v2:172.21.15.176:6812/3003473398 conn(0x7f319c03cf20 0x7f319c03f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f31aeffd640 10 monclient: get_auth_request con 0x7f319c03cf20 auth_method 0 2024-09-10T20:26:23.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 librados: starting msgr at 2024-09-10T20:26:23.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 librados: starting objecter 2024-09-10T20:26:23.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 1 -- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 msgr2=0x7f77a81ddf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f779ffff640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a81ddf80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f779000c3e0 tx=0x7f779000c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 client.?.objecter ms_handle_connect 0x7f77a809fa10 2024-09-10T20:26:23.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:23.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 Processor -- start 2024-09-10T20:26:23.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 -- start start 2024-09-10T20:26:23.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 librados: setting wanted keys 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 1 -- 172.21.15.153:0/552200403 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77900119a0 con 0x7f77a809fa10 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 monclient: dump: 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.411 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 1 -- 172.21.15.153:0/552200403 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7790017070 con 0x7f77a809fa10 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 1 -- 172.21.15.153:0/552200403 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7790011d80 con 0x7f77a809fa10 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a57fa640 10 monclient: dump: 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.412 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.413 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 1 librados: calling monclient init 2024-09-10T20:26:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 10 monclient: init 2024-09-10T20:26:23.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7fe45c92c640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 5 monclient: authenticate success, global_id 4268 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 10 monclient: _renew_subs 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77a80a1960 con 0x7f77a809fa10 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 10 client.4268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 10 monclient: _renew_subs 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f77a8137970 con 0x7f77a809fa10 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 1 librados: init done 2024-09-10T20:26:23.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77a6ffd640 4 set_mon_vals no callback set 2024-09-10T20:26:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.309+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f778c005090 con 0x7f77a809fa10 2024-09-10T20:26:23.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 0x7fe4581ce440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 10 monclient(hunting): picked mon.noname-c con 0x7fe4580a2240 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580a2620 0x7fe4581358c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 10 monclient(hunting): picked mon.noname-b con 0x7fe4580a2620 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4581366a0 0x7fe458135e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 0x7fe4581ce440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 0x7fe4581ce440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55714/0 (socket says 172.21.15.153:55714) 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe455d9b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580a2620 0x7fe4581358c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe455d9b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580a2620 0x7fe4581358c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.153:40016/0 (socket says 172.21.15.153:40016) 2024-09-10T20:26:23.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4581366a0 0x7fe458135e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f77a57fa640 1 -- 172.21.15.153:0/552200403 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f779001c460 con 0x7f77a809fa10 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f77a57fa640 1 --2- 172.21.15.153:0/552200403 >> v2:172.21.15.176:6812/3003473398 conn(0x7f778803cef0 0x7f778803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f77a57fa640 1 -- 172.21.15.153:0/552200403 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f779004fb20 con 0x7f77a809fa10 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f77a57fa640 10 client.4268.objecter ms_dispatch 0x7f77a8000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f77a57fa640 3 client.4268.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f77a57fa640 3 client.4268.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f77a57fa640 20 client.4268.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4581366a0 0x7fe458135e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42322/0 (socket says 172.21.15.153:42322) 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 10 monclient(hunting): picked mon.noname-a con 0x7fe4581366a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:23.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 1 -- 172.21.15.153:0/1554540095 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe4580a30e0 con 0x7fe4581366a0 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 -- 172.21.15.153:0/1554540095 learned_addr learned my addr 172.21.15.153:0/1554540095 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 1 -- 172.21.15.153:0/1554540095 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe4580a2f60 con 0x7fe4580a2620 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 1 -- 172.21.15.153:0/1554540095 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe458140180 con 0x7fe4580a2240 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45c92c640 10 monclient(hunting): authenticate will time out at 943.457001s 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient(hunting): get_auth_request con 0x7fe4580a2240 auth_method 0 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe455d9b640 10 monclient(hunting): get_auth_request con 0x7fe4580a2620 auth_method 0 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe455d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe455d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe455d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 10 monclient(hunting): get_auth_request con 0x7fe4581366a0 auth_method 0 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe455d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:23.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe455d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe455d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient(hunting): handle_auth_done global_id 4271 payload 824 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 -- 172.21.15.153:0/1554540095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580a2620 msgr2=0x7fe4581358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 --2- 172.21.15.153:0/1554540095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580a2620 0x7fe4581358c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 -- 172.21.15.153:0/1554540095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4581366a0 msgr2=0x7fe458135e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 --2- 172.21.15.153:0/1554540095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4581366a0 0x7fe458135e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient: _finish_hunting 0 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 monclient: found mon.noname-c 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 -- 172.21.15.153:0/1554540095 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4580a4ac0 con 0x7fe4580a2240 2024-09-10T20:26:23.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe456d9d640 1 --2- 172.21.15.153:0/1554540095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4581366a0 0x7fe458135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:23.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f31977fe640 10 client.4263.objecter ms_handle_connect 0x7f319c03cf20 2024-09-10T20:26:23.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f31aeffd640 1 --2- 172.21.15.153:0/1442100865 >> v2:172.21.15.176:6812/3003473398 conn(0x7f319c03cf20 0x7f319c03f3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f31a00ab000 tx=0x7f31a00a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f77a7fff640 1 --2- 172.21.15.153:0/552200403 >> v2:172.21.15.176:6812/3003473398 conn(0x7f778803cef0 0x7f778803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7f77a7fff640 10 monclient: get_auth_request con 0x7f778803cef0 auth_method 0 2024-09-10T20:26:23.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 -- 172.21.15.153:0/1554540095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 msgr2=0x7fe4581ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:23.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe45659c640 1 --2- 172.21.15.153:0/1554540095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 0x7fe4581ce440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe44800b550 tx=0x7fe44800ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe446ffd640 10 client.?.objecter ms_handle_connect 0x7fe4580a2240 2024-09-10T20:26:23.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.313+0000 7fe446ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:23.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.317+0000 7f77a7fff640 1 --2- 172.21.15.153:0/552200403 >> v2:172.21.15.176:6812/3003473398 conn(0x7f778803cef0 0x7f778803f3b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f779800b800 tx=0x7f779800b3a0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.317+0000 7f77a57fa640 10 client.4268.objecter ms_handle_connect 0x7f778803cef0 2024-09-10T20:26:23.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.317+0000 7f31977fe640 1 -- 172.21.15.153:0/1442100865 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f31a401ae30 con 0x7f31b00a4580 2024-09-10T20:26:23.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.317+0000 7f31977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.317+0000 7f31977fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:23.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 1 -- 172.21.15.153:0/1554540095 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe448014070 con 0x7fe4580a2240 2024-09-10T20:26:23.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:23.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 10 monclient: dump: 2024-09-10T20:26:23.424 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.424 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.424 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:23.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 1 -- 172.21.15.153:0/1554540095 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe448010aa0 con 0x7fe4580a2240 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 1 -- 172.21.15.153:0/1554540095 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44800f3c0 con 0x7fe4580a2240 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe446ffd640 10 monclient: dump: 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:23.426 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe454d99640 4 set_mon_vals no callback set 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe45c92c640 5 monclient: authenticate success, global_id 4271 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe45c92c640 10 monclient: _renew_subs 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe45c92c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe45c92c640 1 -- 172.21.15.153:0/1554540095 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe45812e330 con 0x7fe4580a2240 2024-09-10T20:26:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7f77a57fa640 1 -- 172.21.15.153:0/552200403 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f779001bd60 con 0x7f77a809fa10 2024-09-10T20:26:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7f77a57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7f77a57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe45c92c640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe45c92c640 10 monclient: _renew_subs 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe45c92c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe45c92c640 1 -- 172.21.15.153:0/1554540095 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe45809e650 con 0x7fe4580a2240 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.321+0000 7fe45c92c640 1 librados: init done 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe444ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe444ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe444ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe444ff9640 1 -- 172.21.15.153:0/1554540095 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe430005090 con 0x7fe4580a2240 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe446ffd640 1 -- 172.21.15.153:0/1554540095 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fe448017070 con 0x7fe4580a2240 2024-09-10T20:26:23.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe446ffd640 1 --2- 172.21.15.153:0/1554540095 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe43803cef0 0x7fe43803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe446ffd640 1 -- 172.21.15.153:0/1554540095 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7fe44804f0b0 con 0x7fe4580a2240 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe446ffd640 10 client.4271.objecter ms_dispatch 0x7fe458000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe446ffd640 3 client.4271.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe446ffd640 3 client.4271.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe446ffd640 20 client.4271.objecter dump_active .. 0 homeless 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe455d9b640 1 --2- 172.21.15.153:0/1554540095 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe43803cef0 0x7fe43803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.325+0000 7fe455d9b640 10 monclient: get_auth_request con 0x7fe43803cef0 auth_method 0 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.329+0000 7fe446ffd640 10 client.4271.objecter ms_handle_connect 0x7fe43803cef0 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.329+0000 7fe455d9b640 1 --2- 172.21.15.153:0/1554540095 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe43803cef0 0x7fe43803f3b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe45812e540 tx=0x7fe44c009110 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.329+0000 7fe446ffd640 1 -- 172.21.15.153:0/1554540095 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fe4580a4ac0 con 0x7fe4580a2240 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.329+0000 7fe446ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:23.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.329+0000 7fe446ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:23.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-10T20:26:23.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-10T20:26:23.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f31b0126980 con 0x7f31b00a4580 2024-09-10T20:26:23.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31977fe640 1 -- 172.21.15.153:0/1442100865 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f31a40253d0 con 0x7f31b00a4580 2024-09-10T20:26:23.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-10T20:26:23.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31977fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:23.492 INFO:teuthology.orchestra.run.smithi153.stdout:55834574852 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 10 librados: watch_flush enter 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 10 librados: watch_flush exit 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 20 client.4263.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 10 client.4263.objecter successfully canceled tick 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 >> v2:172.21.15.176:6812/3003473398 conn(0x7f319c03cf20 msgr2=0x7f319c03f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 --2- 172.21.15.153:0/1442100865 >> v2:172.21.15.176:6812/3003473398 conn(0x7f319c03cf20 0x7f319c03f3e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f31a00ab000 tx=0x7f31a00a6000 comp rx=0 tx=0).stop 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 10 monclient: shutdown 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b00a4580 msgr2=0x7f31b0125960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b00a4580 0x7f31b0125960 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f31a400c450 tx=0x7f31a400c910 comp rx=0 tx=0).stop 2024-09-10T20:26:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 shutdown_connections 2024-09-10T20:26:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 --2- 172.21.15.153:0/1442100865 >> v2:172.21.15.176:6812/3003473398 conn(0x7f319c03cf20 0x7f319c03f3e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f31b00a49a0 0x7f31b013c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f31b00a4580 0x7f31b0125960 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 --2- 172.21.15.153:0/1442100865 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f31b00a41a0 0x7f31b0123940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 >> 172.21.15.153:0/1442100865 conn(0x7f31b00ab7e0 msgr2=0x7f31b00a00c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 shutdown_connections 2024-09-10T20:26:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 -- 172.21.15.153:0/1442100865 wait complete. 2024-09-10T20:26:23.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.485+0000 7f31b549b640 1 librados: shutdown 2024-09-10T20:26:23.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.493+0000 7f77ae502640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-10T20:26:23.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.493+0000 7f77ae502640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-10T20:26:23.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.493+0000 7f77ae502640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.493+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f778c005650 con 0x7f77a809fa10 2024-09-10T20:26:23.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77a57fa640 1 -- 172.21.15.153:0/552200403 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f779001a360 con 0x7f77a809fa10 2024-09-10T20:26:23.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77a57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-10T20:26:23.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77a57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:23.503 INFO:teuthology.orchestra.run.smithi153.stdout:51539607557 2024-09-10T20:26:23.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 10 librados: watch_flush enter 2024-09-10T20:26:23.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 10 librados: watch_flush exit 2024-09-10T20:26:23.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 20 client.4268.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 10 client.4268.objecter successfully canceled tick 2024-09-10T20:26:23.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 >> v2:172.21.15.176:6812/3003473398 conn(0x7f778803cef0 msgr2=0x7f778803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 --2- 172.21.15.153:0/552200403 >> v2:172.21.15.176:6812/3003473398 conn(0x7f778803cef0 0x7f778803f3b0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f779800b800 tx=0x7f779800b3a0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 10 monclient: shutdown 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 msgr2=0x7f77a81ddf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a81ddf80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f779000c3e0 tx=0x7f779000c8a0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 shutdown_connections 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f77a80a37d0 0x7f77a8133360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 --2- 172.21.15.153:0/552200403 >> v2:172.21.15.176:6812/3003473398 conn(0x7f778803cef0 0x7f778803f3b0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f77a809fa10 0x7f77a81ddf80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 --2- 172.21.15.153:0/552200403 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f77a809cfb0 0x7f77a81dda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 >> 172.21.15.153:0/552200403 conn(0x7f77a80a8480 msgr2=0x7f77a809ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 shutdown_connections 2024-09-10T20:26:23.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 -- 172.21.15.153:0/552200403 wait complete. 2024-09-10T20:26:23.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.497+0000 7f77ae502640 1 librados: shutdown 2024-09-10T20:26:23.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.501+0000 7fe444ff9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-10T20:26:23.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.501+0000 7fe444ff9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-10T20:26:23.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.501+0000 7fe444ff9640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:23.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.501+0000 7fe444ff9640 1 -- 172.21.15.153:0/1554540095 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fe430005380 con 0x7fe4580a2240 2024-09-10T20:26:23.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.501+0000 7fe446ffd640 1 -- 172.21.15.153:0/1554540095 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe44800fd10 con 0x7fe4580a2240 2024-09-10T20:26:23.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.501+0000 7fe446ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-10T20:26:23.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.501+0000 7fe446ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:23.511 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.1 2024-09-10T20:26:23.511 INFO:teuthology.orchestra.run.smithi153.stdout:47244640261 2024-09-10T20:26:23.511 DEBUG:teuthology.parallel:result is None 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 10 librados: watch_flush enter 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 10 librados: watch_flush exit 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 20 client.4271.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 10 client.4271.objecter successfully canceled tick 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 -- 172.21.15.153:0/1554540095 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe43803cef0 msgr2=0x7fe43803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 --2- 172.21.15.153:0/1554540095 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe43803cef0 0x7fe43803f3b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe45812e540 tx=0x7fe44c009110 comp rx=0 tx=0).stop 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 10 monclient: shutdown 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 -- 172.21.15.153:0/1554540095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 msgr2=0x7fe4581ce440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 --2- 172.21.15.153:0/1554540095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 0x7fe4581ce440 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe44800b550 tx=0x7fe44800ba10 comp rx=0 tx=0).stop 2024-09-10T20:26:23.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 -- 172.21.15.153:0/1554540095 shutdown_connections 2024-09-10T20:26:23.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 --2- 172.21.15.153:0/1554540095 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe4581366a0 0x7fe458135e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 --2- 172.21.15.153:0/1554540095 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe4580a2620 0x7fe4581358c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 --2- 172.21.15.153:0/1554540095 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe43803cef0 0x7fe43803f3b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 --2- 172.21.15.153:0/1554540095 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe4580a2240 0x7fe4581ce440 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 -- 172.21.15.153:0/1554540095 >> 172.21.15.153:0/1554540095 conn(0x7fe4580a7830 msgr2=0x7fe4580725f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 -- 172.21.15.153:0/1554540095 shutdown_connections 2024-09-10T20:26:23.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 -- 172.21.15.153:0/1554540095 wait complete. 2024-09-10T20:26:23.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.505+0000 7fe444ff9640 1 librados: shutdown 2024-09-10T20:26:23.524 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.4 2024-09-10T20:26:23.524 DEBUG:teuthology.parallel:result is None 2024-09-10T20:26:23.530 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.5 2024-09-10T20:26:23.530 DEBUG:teuthology.parallel:result is None 2024-09-10T20:26:23.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-10T20:26:23.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-10T20:26:23.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 -- 172.21.15.153:0/2541657017 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f92581217c0 con 0x7f92580a49d0 2024-09-10T20:26:23.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923ffff640 1 -- 172.21.15.153:0/2541657017 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f92500c3360 con 0x7f92580a49d0 2024-09-10T20:26:23.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-10T20:26:23.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:23.588 INFO:teuthology.orchestra.run.smithi153.stdout:51539607557 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 10 librados: watch_flush enter 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 10 librados: watch_flush exit 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 20 client.4254.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 10 client.4254.objecter successfully canceled tick 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 -- 172.21.15.153:0/2541657017 >> v2:172.21.15.176:6812/3003473398 conn(0x7f924403cef0 msgr2=0x7f924403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 --2- 172.21.15.153:0/2541657017 >> v2:172.21.15.176:6812/3003473398 conn(0x7f924403cef0 0x7f924403f3b0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9248007ca0 tx=0x7f9248007bf0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 10 monclient: shutdown 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 -- 172.21.15.153:0/2541657017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92580a49d0 msgr2=0x7f925813c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 --2- 172.21.15.153:0/2541657017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92580a49d0 0x7f925813c9a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f925001c680 tx=0x7f92500195f0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 -- 172.21.15.153:0/2541657017 shutdown_connections 2024-09-10T20:26:23.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 --2- 172.21.15.153:0/2541657017 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92580a49d0 0x7f925813c9a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 --2- 172.21.15.153:0/2541657017 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92580a4580 0x7f92581375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 --2- 172.21.15.153:0/2541657017 >> v2:172.21.15.176:6812/3003473398 conn(0x7f924403cef0 0x7f924403f3b0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 --2- 172.21.15.153:0/2541657017 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92580a41a0 0x7f92581370a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 -- 172.21.15.153:0/2541657017 >> 172.21.15.153:0/2541657017 conn(0x7f92580ab7e0 msgr2=0x7f925813a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 -- 172.21.15.153:0/2541657017 shutdown_connections 2024-09-10T20:26:23.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 -- 172.21.15.153:0/2541657017 wait complete. 2024-09-10T20:26:23.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.581+0000 7f923dffb640 1 librados: shutdown 2024-09-10T20:26:23.605 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.3 2024-09-10T20:26:23.605 DEBUG:teuthology.parallel:result is None 2024-09-10T20:26:23.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-10T20:26:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-10T20:26:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 -- 172.21.15.153:0/1974313033 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f253809ed70 con 0x7f25381322c0 2024-09-10T20:26:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f252dffb640 1 -- 172.21.15.153:0/1974313033 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2530047d90 con 0x7f25381322c0 2024-09-10T20:26:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f252dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-10T20:26:23.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f252dffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:23.611 INFO:teuthology.orchestra.run.smithi153.stdout:55834574851 2024-09-10T20:26:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 10 librados: watch_flush enter 2024-09-10T20:26:23.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 10 librados: watch_flush exit 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 20 client.4260.objecter shutdown clearing up homeless session... 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 10 client.4260.objecter successfully canceled tick 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 -- 172.21.15.153:0/1974313033 >> v2:172.21.15.176:6812/3003473398 conn(0x7f251c03cef0 msgr2=0x7f251c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 --2- 172.21.15.153:0/1974313033 >> v2:172.21.15.176:6812/3003473398 conn(0x7f251c03cef0 0x7f251c03f3b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f25340097f0 tx=0x7f2534008040 comp rx=0 tx=0).stop 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 10 monclient: shutdown 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 -- 172.21.15.153:0/1974313033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25381322c0 msgr2=0x7f2538132750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 --2- 172.21.15.153:0/1974313033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25381322c0 0x7f2538132750 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f253000caa0 tx=0x7f253000cf60 comp rx=0 tx=0).stop 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 -- 172.21.15.153:0/1974313033 shutdown_connections 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 --2- 172.21.15.153:0/1974313033 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f25381322c0 0x7f2538132750 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 --2- 172.21.15.153:0/1974313033 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f253809fa10 0x7f25381de4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 --2- 172.21.15.153:0/1974313033 >> v2:172.21.15.176:6812/3003473398 conn(0x7f251c03cef0 0x7f251c03f3b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 --2- 172.21.15.153:0/1974313033 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f253809cfb0 0x7f25381ddf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:23.614 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 -- 172.21.15.153:0/1974313033 >> 172.21.15.153:0/1974313033 conn(0x7f25380a8480 msgr2=0x7f2538076a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:23.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 -- 172.21.15.153:0/1974313033 shutdown_connections 2024-09-10T20:26:23.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 -- 172.21.15.153:0/1974313033 wait complete. 2024-09-10T20:26:23.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:23.605+0000 7f251b7fe640 1 librados: shutdown 2024-09-10T20:26:23.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574851 got 55834574851 for osd.0 2024-09-10T20:26:23.628 DEBUG:teuthology.parallel:result is None 2024-09-10T20:26:24.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.093+0000 7f14f2ffd640 1 -- 172.21.15.153:0/2085301817 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f14fc12d990 con 0x7f14fc13fca0 2024-09-10T20:26:24.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.093+0000 7f14f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:24.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.093+0000 7f14f2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.257+0000 7f1500b02640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-10T20:26:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.257+0000 7f1500b02640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-10T20:26:24.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.257+0000 7f1500b02640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.257+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f14fc12ca30 con 0x7f14fc13fca0 2024-09-10T20:26:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.257+0000 7f14f2ffd640 1 -- 172.21.15.153:0/2085301817 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f14f4010040 con 0x7f14fc13fca0 2024-09-10T20:26:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f14f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-10T20:26:24.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f14f2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:24.266 INFO:teuthology.orchestra.run.smithi153.stdout:55834574852 2024-09-10T20:26:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 10 librados: watch_flush enter 2024-09-10T20:26:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 10 librados: watch_flush exit 2024-09-10T20:26:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 20 client.4240.objecter shutdown clearing up homeless session... 2024-09-10T20:26:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 10 client.4240.objecter successfully canceled tick 2024-09-10T20:26:24.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 >> v2:172.21.15.176:6812/3003473398 conn(0x7f14dc03ce50 msgr2=0x7f14dc03f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 --2- 172.21.15.153:0/2085301817 >> v2:172.21.15.176:6812/3003473398 conn(0x7f14dc03ce50 0x7f14dc03f310 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f14ec007fd0 tx=0x7f14ec007e40 comp rx=0 tx=0).stop 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 10 monclient: shutdown 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14fc13fca0 msgr2=0x7f14fc1d4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14fc13fca0 0x7f14fc1d4710 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14f400efc0 tx=0x7f14f400c490 comp rx=0 tx=0).stop 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 shutdown_connections 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f14fc13fca0 0x7f14fc1d4710 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 --2- 172.21.15.153:0/2085301817 >> v2:172.21.15.176:6812/3003473398 conn(0x7f14dc03ce50 0x7f14dc03f310 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f14fc0a49a0 0x7f14fc13f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 --2- 172.21.15.153:0/2085301817 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f14fc0a41a0 0x7f14fc073fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 >> 172.21.15.153:0/2085301817 conn(0x7f14fc0ab7e0 msgr2=0x7f14fc09f520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 shutdown_connections 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 -- 172.21.15.153:0/2085301817 wait complete. 2024-09-10T20:26:24.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.261+0000 7f1500b02640 1 librados: shutdown 2024-09-10T20:26:24.284 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574852 got 55834574852 for osd.2 2024-09-10T20:26:24.284 DEBUG:teuthology.parallel:result is None 2024-09-10T20:26:24.284 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:26:24.285 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:24.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.389+0000 7fda51004640 10 monclient: get_monmap_and_config 2024-09-10T20:26:24.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.389+0000 7fda51004640 10 monclient: build_initial_monmap 2024-09-10T20:26:24.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.389+0000 7fda51004640 10 monclient: monmap: 2024-09-10T20:26:24.398 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:24.398 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.398 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:24.397434+0000 2024-09-10T20:26:24.399 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:24.397434+0000 2024-09-10T20:26:24.399 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:24.399 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.399 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:24.399 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:24.399 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:24.399 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:24.399 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:24.399 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 1 Processor -- start 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 1 -- start start 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 10 monclient: init 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 10 monclient: _add_conns ranks=[2,1,0,4,3] 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c0a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 10 monclient(hunting): picked mon.noname-b con 0x7fda4c0a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda4c0a4580 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda4c0a49a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4c0a67f0 con 0x7fda4c0a49a0 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4c0a6a20 con 0x7fda4c0a4580 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda4c0a2570 con 0x7fda4c0a41a0 2024-09-10T20:26:24.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda51004640 10 monclient(hunting): authenticate will time out at 944.537183s 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4b59e640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fda4c0a49a0 0x7fda4c0a60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60492/0 (socket says 172.21.15.153:60492) 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4b59e640 1 -- 172.21.15.153:0/3895180461 learned_addr learned my addr 172.21.15.153:0/3895180461 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 1 --2- 172.21.15.153:0/3895180461 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c0a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient(hunting): get_auth_request con 0x7fda4c0a41a0 auth_method 0 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient(hunting): handle_auth_done global_id 4274 payload 274 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 1 -- 172.21.15.153:0/3895180461 >> v1:172.21.15.176:6789/0 conn(0x7fda4c0a4580 legacy=0x7fda4c0a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:24.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 1 -- 172.21.15.153:0/3895180461 >> v1:172.21.15.153:6789/0 conn(0x7fda4c0a49a0 legacy=0x7fda4c0a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient: _finish_hunting 0 2024-09-10T20:26:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 1 monclient: found mon.noname-b 2024-09-10T20:26:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:24.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.393+0000 7fda4ad9d640 1 -- 172.21.15.153:0/3895180461 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c0a3c80 con 0x7fda4c0a41a0 2024-09-10T20:26:24.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda4ad9d640 1 -- 172.21.15.153:0/3895180461 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 msgr2=0x7fda4c0a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:24.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda4ad9d640 1 --2- 172.21.15.153:0/3895180461 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c0a3740 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda44009dc0 tx=0x7fda4401c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdfa91a9a2fae934 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:24.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 1 -- 172.21.15.153:0/3895180461 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda440ad070 con 0x7fda4c0a41a0 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 10 monclient: dump: 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:24.408 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 10 monclient: _finish_auth 0 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 10 monclient: _check_auth_tickets 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 1 -- 172.21.15.153:0/3895180461 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda440a9710 con 0x7fda4c0a41a0 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 1 -- 172.21.15.153:0/3895180461 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda440a9ae0 con 0x7fda4c0a41a0 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 5 monclient: authenticate success, global_id 4274 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 4 set_mon_vals no callback set 2024-09-10T20:26:24.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda49d9b640 10 monclient: dump: 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:24.410 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 10 monclient: shutdown 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 -- 172.21.15.153:0/3895180461 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 msgr2=0x7fda4c0a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 --2- 172.21.15.153:0/3895180461 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c0a3740 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fda44009dc0 tx=0x7fda4401c850 comp rx=0 tx=0).stop 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 -- 172.21.15.153:0/3895180461 shutdown_connections 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 --2- 172.21.15.153:0/3895180461 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c0a3740 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 -- 172.21.15.153:0/3895180461 >> 172.21.15.153:0/3895180461 conn(0x7fda4c0ab7e0 msgr2=0x7fda4c0a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 -- 172.21.15.153:0/3895180461 shutdown_connections 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 -- 172.21.15.153:0/3895180461 wait complete. 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 10 monclient: build_initial_monmap 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 10 monclient: monmap: 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:24.411 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 librados: starting msgr at 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 librados: starting objecter 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 Processor -- start 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 -- start start 2024-09-10T20:26:24.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 librados: setting wanted keys 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 librados: calling monclient init 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 10 monclient: init 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c128aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 10 monclient(hunting): picked mon.noname-c con 0x7fda4c0a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda4c0a4580 0x7fda4c140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.397+0000 7fda51004640 10 monclient(hunting): picked mon.noname-b con 0x7fda4c0a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda4c0a49a0 0x7fda4c1d0a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient(hunting): picked mon.noname-a con 0x7fda4c0a49a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c144630 con 0x7fda4c0a49a0 2024-09-10T20:26:24.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda4c070400 con 0x7fda4c0a4580 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda4c1447b0 con 0x7fda4c0a41a0 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient(hunting): authenticate will time out at 944.544035s 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c128aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c128aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55730/0 (socket says 172.21.15.153:55730) 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 1 -- 172.21.15.153:0/4144739035 learned_addr learned my addr 172.21.15.153:0/4144739035 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 10 monclient(hunting): get_auth_request con 0x7fda4c0a41a0 auth_method 0 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 1 --2- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda4c0a49a0 0x7fda4c1d0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:24.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient(hunting): get_auth_request con 0x7fda4c0a49a0 auth_method 0 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:24.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient(hunting): handle_auth_done global_id 4272 payload 824 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 1 -- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 msgr2=0x7fda4c128aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 1 --2- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c128aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 1 -- 172.21.15.153:0/4144739035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda4c0a4580 msgr2=0x7fda4c140920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 1 --2- 172.21.15.153:0/4144739035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda4c0a4580 0x7fda4c140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient: _finish_hunting 0 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 1 monclient: found mon.noname-a 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 1 -- 172.21.15.153:0/4144739035 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c0a63a0 con 0x7fda4c0a49a0 2024-09-10T20:26:24.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4ad9d640 1 --2- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c128aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:24.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 1 -- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda4c0a49a0 msgr2=0x7fda4c1d0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:24.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4b59e640 1 --2- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda4c0a49a0 0x7fda4c1d0a80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00c410 tx=0x7fda3c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:24.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 client.?.objecter ms_handle_connect 0x7fda4c0a49a0 2024-09-10T20:26:24.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:24.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 1 -- 172.21.15.153:0/4144739035 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c017070 con 0x7fda4c0a49a0 2024-09-10T20:26:24.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:24.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:24.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 monclient: dump: 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:24.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 1 -- 172.21.15.153:0/4144739035 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda3c0070d0 con 0x7fda4c0a49a0 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 5 monclient: authenticate success, global_id 4272 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient: _renew_subs 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda4c12a2a0 con 0x7fda4c0a49a0 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 1 -- 172.21.15.153:0/4144739035 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c007500 con 0x7fda4c0a49a0 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda437fe640 10 monclient: dump: 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:24.421 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda4959a640 4 set_mon_vals no callback set 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient: _renew_subs 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:24.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4c129430 con 0x7fda4c0a49a0 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 1 librados: init done 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.401+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda4c0a3740 con 0x7fda4c0a49a0 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.405+0000 7fda437fe640 1 -- 172.21.15.153:0/4144739035 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fda3c01a070 con 0x7fda4c0a49a0 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.405+0000 7fda437fe640 1 --2- 172.21.15.153:0/4144739035 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda2c03cea0 0x7fda2c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.405+0000 7fda437fe640 1 -- 172.21.15.153:0/4144739035 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7fda3c04ea70 con 0x7fda4c0a49a0 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.405+0000 7fda437fe640 10 client.4272.objecter ms_dispatch 0x7fda4c000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.405+0000 7fda437fe640 3 client.4272.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:24.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.405+0000 7fda437fe640 3 client.4272.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:24.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.405+0000 7fda437fe640 20 client.4272.objecter dump_active .. 0 homeless 2024-09-10T20:26:24.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.405+0000 7fda4ad9d640 1 --2- 172.21.15.153:0/4144739035 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda2c03cea0 0x7fda2c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:24.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.405+0000 7fda4ad9d640 10 monclient: get_auth_request con 0x7fda2c03cea0 auth_method 0 2024-09-10T20:26:24.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.409+0000 7fda437fe640 10 client.4272.objecter ms_handle_connect 0x7fda2c03cea0 2024-09-10T20:26:24.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.409+0000 7fda4ad9d640 1 --2- 172.21.15.153:0/4144739035 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda2c03cea0 0x7fda2c03f360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fda44009bd0 tx=0x7fda440b5040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:24.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.409+0000 7fda437fe640 1 -- 172.21.15.153:0/4144739035 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fda3c021020 con 0x7fda4c0a49a0 2024-09-10T20:26:24.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.409+0000 7fda437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:24.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.409+0000 7fda437fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:24.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.561+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fda4c127f40 con 0x7fda2c03cea0 2024-09-10T20:26:24.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.561+0000 7fda437fe640 1 -- 172.21.15.153:0/4144739035 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34213 (secure 0 0 0) 0x7fda4c127f40 con 0x7fda2c03cea0 2024-09-10T20:26:24.570 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:24.571 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-10T20:26:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 10 librados: watch_flush enter 2024-09-10T20:26:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 10 librados: watch_flush exit 2024-09-10T20:26:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 20 client.4272.objecter shutdown clearing up homeless session... 2024-09-10T20:26:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 10 client.4272.objecter successfully canceled tick 2024-09-10T20:26:24.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda2c03cea0 msgr2=0x7fda2c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 --2- 172.21.15.153:0/4144739035 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda2c03cea0 0x7fda2c03f360 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fda44009bd0 tx=0x7fda440b5040 comp rx=0 tx=0).stop 2024-09-10T20:26:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 10 monclient: shutdown 2024-09-10T20:26:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda4c0a49a0 msgr2=0x7fda4c1d0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 --2- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda4c0a49a0 0x7fda4c1d0a80 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fda3c00c410 tx=0x7fda3c00c8d0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 shutdown_connections 2024-09-10T20:26:24.576 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 --2- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda4c0a49a0 0x7fda4c1d0a80 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 --2- 172.21.15.153:0/4144739035 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda4c0a4580 0x7fda4c140920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 --2- 172.21.15.153:0/4144739035 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda2c03cea0 0x7fda2c03f360 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 --2- 172.21.15.153:0/4144739035 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0a41a0 0x7fda4c128aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 >> 172.21.15.153:0/4144739035 conn(0x7fda4c0ab7e0 msgr2=0x7fda4c09fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:24.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 shutdown_connections 2024-09-10T20:26:24.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 -- 172.21.15.153:0/4144739035 wait complete. 2024-09-10T20:26:24.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.565+0000 7fda51004640 1 librados: shutdown 2024-09-10T20:26:24.587 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-10T20:26:23.069400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163316,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067724,"statfs":{"total":579820584960,"available":579653349376,"internally_reserved":0,"allocated":1163264,"data_stored":1383940,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.723438"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773251+0000","last_change":"2024-09-10T20:26:21.773251+0000","last_active":"2024-09-10T20:26:21.773251+0000","last_peered":"2024-09-10T20:26:21.773251+0000","last_clean":"2024-09-10T20:26:21.773251+0000","last_became_active":"2024-09-10T20:26:19.356844+0000","last_became_peered":"2024-09-10T20:26:19.356844+0000","last_unstale":"2024-09-10T20:26:21.773251+0000","last_undegraded":"2024-09-10T20:26:21.773251+0000","last_fullsized":"2024-09-10T20:26:21.773251+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:23.413224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.381255+0000","last_change":"2024-09-10T20:26:21.381255+0000","last_active":"2024-09-10T20:26:21.381255+0000","last_peered":"2024-09-10T20:26:21.381255+0000","last_clean":"2024-09-10T20:26:21.381255+0000","last_became_active":"2024-09-10T20:26:19.363220+0000","last_became_peered":"2024-09-10T20:26:19.363220+0000","last_unstale":"2024-09-10T20:26:21.381255+0000","last_undegraded":"2024-09-10T20:26:21.381255+0000","last_fullsized":"2024-09-10T20:26:21.381255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:09.279291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096679600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773020+0000","last_change":"2024-09-10T20:26:21.773020+0000","last_active":"2024-09-10T20:26:21.773020+0000","last_peered":"2024-09-10T20:26:21.773020+0000","last_clean":"2024-09-10T20:26:21.773020+0000","last_became_active":"2024-09-10T20:26:19.363294+0000","last_became_peered":"2024-09-10T20:26:19.363294+0000","last_unstale":"2024-09-10T20:26:21.773020+0000","last_undegraded":"2024-09-10T20:26:21.773020+0000","last_fullsized":"2024-09-10T20:26:21.773020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:54.300619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053952999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.378377+0000","last_change":"2024-09-10T20:26:21.378377+0000","last_active":"2024-09-10T20:26:21.378377+0000","last_peered":"2024-09-10T20:26:21.378377+0000","last_clean":"2024-09-10T20:26:21.378377+0000","last_became_active":"2024-09-10T20:26:19.359961+0000","last_became_peered":"2024-09-10T20:26:19.359961+0000","last_unstale":"2024-09-10T20:26:21.378377+0000","last_undegraded":"2024-09-10T20:26:21.378377+0000","last_fullsized":"2024-09-10T20:26:21.378377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:21:01.940376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.394496+0000","last_change":"2024-09-10T20:26:21.386697+0000","last_active":"2024-09-10T20:26:21.394496+0000","last_peered":"2024-09-10T20:26:21.394496+0000","last_clean":"2024-09-10T20:26:21.394496+0000","last_became_active":"2024-09-10T20:26:19.360427+0000","last_became_peered":"2024-09-10T20:26:19.360427+0000","last_unstale":"2024-09-10T20:26:21.394496+0000","last_undegraded":"2024-09-10T20:26:21.394496+0000","last_fullsized":"2024-09-10T20:26:21.394496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:53.136006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020581079999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.575187+0000","last_change":"2024-09-10T20:26:21.575187+0000","last_active":"2024-09-10T20:26:21.575187+0000","last_peered":"2024-09-10T20:26:21.575187+0000","last_clean":"2024-09-10T20:26:21.575187+0000","last_became_active":"2024-09-10T20:26:19.362858+0000","last_became_peered":"2024-09-10T20:26:19.362858+0000","last_unstale":"2024-09-10T20:26:21.575187+0000","last_undegraded":"2024-09-10T20:26:21.575187+0000","last_fullsized":"2024-09-10T20:26:21.575187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:22:17.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078714099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.378915+0000","last_change":"2024-09-10T20:26:21.378915+0000","last_active":"2024-09-10T20:26:21.378915+0000","last_peered":"2024-09-10T20:26:21.378915+0000","last_clean":"2024-09-10T20:26:21.378915+0000","last_became_active":"2024-09-10T20:26:19.362158+0000","last_became_peered":"2024-09-10T20:26:19.362158+0000","last_unstale":"2024-09-10T20:26:21.378915+0000","last_undegraded":"2024-09-10T20:26:21.378915+0000","last_fullsized":"2024-09-10T20:26:21.378915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:50.708554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010122729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773025+0000","last_change":"2024-09-10T20:26:21.773025+0000","last_active":"2024-09-10T20:26:21.773025+0000","last_peered":"2024-09-10T20:26:21.773025+0000","last_clean":"2024-09-10T20:26:21.773025+0000","last_became_active":"2024-09-10T20:26:19.359775+0000","last_became_peered":"2024-09-10T20:26:19.359775+0000","last_unstale":"2024-09-10T20:26:21.773025+0000","last_undegraded":"2024-09-10T20:26:21.773025+0000","last_fullsized":"2024-09-10T20:26:21.773025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:35.427907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054594699999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.772391+0000","last_change":"2024-09-10T20:26:14.579551+0000","last_active":"2024-09-10T20:26:21.772391+0000","last_peered":"2024-09-10T20:26:21.772391+0000","last_clean":"2024-09-10T20:26:21.772391+0000","last_became_active":"2024-09-10T20:26:14.577602+0000","last_became_peered":"2024-09-10T20:26:14.577602+0000","last_unstale":"2024-09-10T20:26:21.772391+0000","last_undegraded":"2024-09-10T20:26:21.772391+0000","last_fullsized":"2024-09-10T20:26:21.772391+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_clean_scrub_stamp":"2024-09-10T20:26:13.287430+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:41.463707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":583269,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":583269,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":54341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:24.588 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:24.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.697+0000 7f6df10cd640 10 monclient: get_monmap_and_config 2024-09-10T20:26:24.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.697+0000 7f6df10cd640 10 monclient: build_initial_monmap 2024-09-10T20:26:24.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.697+0000 7f6df10cd640 10 monclient: monmap: 2024-09-10T20:26:24.705 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:24.705 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.705 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:24.704563+0000 2024-09-10T20:26:24.705 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:24.704563+0000 2024-09-10T20:26:24.706 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:24.706 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.706 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:24.706 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:24.706 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:24.706 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:24.706 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:24.706 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.697+0000 7f6df10cd640 1 Processor -- start 2024-09-10T20:26:24.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.697+0000 7f6df10cd640 1 -- start start 2024-09-10T20:26:24.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.697+0000 7f6df10cd640 10 monclient: init 2024-09-10T20:26:24.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 10 monclient: _add_conns ranks=[1,0,2,3,4] 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6dec06d5d0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6dec0a2770 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec0a2b80 0x7f6dec141160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 10 monclient(hunting): picked mon.noname-b con 0x7f6dec0a2b80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dec074880 con 0x7f6dec0a2770 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dec09f8d0 con 0x7f6dec06d5d0 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6dec09eae0 con 0x7f6dec0a2b80 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6df10cd640 10 monclient(hunting): authenticate will time out at 944.844141s 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec0a2b80 0x7f6dec141160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:24.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6dea59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f6dec0a2770 0x7f6dec13da30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60508/0 (socket says 172.21.15.153:60508) 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6dea59c640 1 -- 172.21.15.153:0/1765895846 learned_addr learned my addr 172.21.15.153:0/1765895846 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient(hunting): get_auth_request con 0x7f6dec0a2b80 auth_method 0 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient(hunting): handle_auth_done global_id 4280 payload 274 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 1 -- 172.21.15.153:0/1765895846 >> v1:172.21.15.176:6789/0 conn(0x7f6dec06d5d0 legacy=0x7f6dec0a2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 1 -- 172.21.15.153:0/1765895846 >> v1:172.21.15.153:6789/0 conn(0x7f6dec0a2770 legacy=0x7f6dec13da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:24.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient: _finish_hunting 0 2024-09-10T20:26:24.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 1 monclient: found mon.noname-b 2024-09-10T20:26:24.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:24.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:24.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.701+0000 7f6deb59e640 1 -- 172.21.15.153:0/1765895846 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec09edc0 con 0x7f6dec0a2b80 2024-09-10T20:26:24.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 1 -- 172.21.15.153:0/1765895846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec0a2b80 msgr2=0x7f6dec141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:24.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 1 --2- 172.21.15.153:0/1765895846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec0a2b80 0x7f6dec141160 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009dc0 tx=0x7f6de401c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20923aa43e47841d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:24.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 1 -- 172.21.15.153:0/1765895846 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de40ac070 con 0x7f6dec0a2b80 2024-09-10T20:26:24.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:24.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:24.718 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 10 monclient: dump: 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.719 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 10 monclient: _finish_auth 0 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 10 monclient: _check_auth_tickets 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 1 -- 172.21.15.153:0/1765895846 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6de40a6720 con 0x7f6dec0a2b80 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 1 -- 172.21.15.153:0/1765895846 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6de40a6af0 con 0x7f6dec0a2b80 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 5 monclient: authenticate success, global_id 4280 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 4 set_mon_vals no callback set 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:24.720 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6de9d9b640 10 monclient: dump: 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.721 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient: shutdown 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 -- 172.21.15.153:0/1765895846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec0a2b80 msgr2=0x7f6dec141160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 --2- 172.21.15.153:0/1765895846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec0a2b80 0x7f6dec141160 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6de4009dc0 tx=0x7f6de401c7f0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 -- 172.21.15.153:0/1765895846 shutdown_connections 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 --2- 172.21.15.153:0/1765895846 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec0a2b80 0x7f6dec141160 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 -- 172.21.15.153:0/1765895846 >> 172.21.15.153:0/1765895846 conn(0x7f6dec0a7830 msgr2=0x7f6dec0a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 -- 172.21.15.153:0/1765895846 shutdown_connections 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 -- 172.21.15.153:0/1765895846 wait complete. 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient: build_initial_monmap 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient: monmap: 2024-09-10T20:26:24.722 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.723 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 librados: starting msgr at 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 librados: starting objecter 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 Processor -- start 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 -- start start 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 librados: setting wanted keys 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 librados: calling monclient init 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient: init 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec06d5d0 0x7f6dec13fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.724 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient(hunting): picked mon.noname-c con 0x7f6dec06d5d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6dec0a2770 0x7f6dec1d0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient(hunting): picked mon.noname-b con 0x7f6dec0a2770 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dec0a2b80 0x7f6dec1d0a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient(hunting): picked mon.noname-a con 0x7f6dec0a2b80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec144250 con 0x7f6dec0a2b80 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6dec1440d0 con 0x7f6dec0a2770 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6dec1443d0 con 0x7f6dec06d5d0 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:24.725 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6df10cd640 10 monclient(hunting): authenticate will time out at 944.851002s 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec06d5d0 0x7f6dec13fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec06d5d0 0x7f6dec13fe80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55746/0 (socket says 172.21.15.153:55746) 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 1 -- 172.21.15.153:0/361829820 learned_addr learned my addr 172.21.15.153:0/361829820 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient(hunting): get_auth_request con 0x7f6dec06d5d0 auth_method 0 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 1 --2- 172.21.15.153:0/361829820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dec0a2b80 0x7f6dec1d0a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 10 monclient(hunting): get_auth_request con 0x7f6dec0a2b80 auth_method 0 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:24.726 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6deb59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient(hunting): handle_auth_done global_id 4283 payload 824 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 1 -- 172.21.15.153:0/361829820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6dec0a2770 msgr2=0x7f6dec1d0520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 1 --2- 172.21.15.153:0/361829820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6dec0a2770 0x7f6dec1d0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 1 -- 172.21.15.153:0/361829820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dec0a2b80 msgr2=0x7f6dec1d0a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 1 --2- 172.21.15.153:0/361829820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dec0a2b80 0x7f6dec1d0a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient: _finish_hunting 0 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 1 monclient: found mon.noname-c 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.705+0000 7f6dead9d640 1 -- 172.21.15.153:0/361829820 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dec090830 con 0x7f6dec06d5d0 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dead9d640 1 -- 172.21.15.153:0/361829820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec06d5d0 msgr2=0x7f6dec13fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dead9d640 1 --2- 172.21.15.153:0/361829820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec06d5d0 0x7f6dec13fe80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00b440 tx=0x7f6ddc00b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 client.?.objecter ms_handle_connect 0x7f6dec06d5d0 2024-09-10T20:26:24.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 1 -- 172.21.15.153:0/361829820 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc015020 con 0x7f6dec06d5d0 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 monclient: dump: 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:24.729 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 1 -- 172.21.15.153:0/361829820 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ddc011a40 con 0x7f6dec06d5d0 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 1 -- 172.21.15.153:0/361829820 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ddc011e30 con 0x7f6dec06d5d0 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 5 monclient: authenticate success, global_id 4283 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 10 monclient: _renew_subs 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:24.730 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dec090b10 con 0x7f6dec06d5d0 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 monclient: dump: 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6de959a640 4 set_mon_vals no callback set 2024-09-10T20:26:24.731 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 10 monclient: _renew_subs 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dec1408c0 con 0x7f6dec06d5d0 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 1 librados: init done 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dec0a2060 con 0x7f6dec06d5d0 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 1 -- 172.21.15.153:0/361829820 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f6ddc018020 con 0x7f6dec06d5d0 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 1 --2- 172.21.15.153:0/361829820 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6dd803cea0 0x7f6dd803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 1 -- 172.21.15.153:0/361829820 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f6ddc04ee50 con 0x7f6dec06d5d0 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 10 client.4283.objecter ms_dispatch 0x7f6dec000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 3 client.4283.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:24.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 3 client.4283.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dd37fe640 20 client.4283.objecter dump_active .. 0 homeless 2024-09-10T20:26:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dea59c640 1 --2- 172.21.15.153:0/361829820 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6dd803cea0 0x7f6dd803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.709+0000 7f6dea59c640 10 monclient: get_auth_request con 0x7f6dd803cea0 auth_method 0 2024-09-10T20:26:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.713+0000 7f6dd37fe640 10 client.4283.objecter ms_handle_connect 0x7f6dd803cea0 2024-09-10T20:26:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.713+0000 7f6dea59c640 1 --2- 172.21.15.153:0/361829820 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6dd803cea0 0x7f6dd803f360 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6dec1d15e0 tx=0x7f6de0007ab0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.713+0000 7f6dd37fe640 1 -- 172.21.15.153:0/361829820 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f6ddc01f030 con 0x7f6dec06d5d0 2024-09-10T20:26:24.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.713+0000 7f6dd37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:24.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.713+0000 7f6dd37fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:24.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.865+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6dec06cd40 con 0x7f6dd803cea0 2024-09-10T20:26:24.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6dd37fe640 1 -- 172.21.15.153:0/361829820 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34213 (secure 0 0 0) 0x7f6dec06cd40 con 0x7f6dd803cea0 2024-09-10T20:26:24.876 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:24.877 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-10T20:26:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 10 librados: watch_flush enter 2024-09-10T20:26:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 10 librados: watch_flush exit 2024-09-10T20:26:24.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 20 client.4283.objecter shutdown clearing up homeless session... 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 10 client.4283.objecter successfully canceled tick 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6dd803cea0 msgr2=0x7f6dd803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 --2- 172.21.15.153:0/361829820 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6dd803cea0 0x7f6dd803f360 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6dec1d15e0 tx=0x7f6de0007ab0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 10 monclient: shutdown 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec06d5d0 msgr2=0x7f6dec13fe80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 --2- 172.21.15.153:0/361829820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec06d5d0 0x7f6dec13fe80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6ddc00b440 tx=0x7f6ddc00b900 comp rx=0 tx=0).stop 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 shutdown_connections 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 --2- 172.21.15.153:0/361829820 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6dec0a2b80 0x7f6dec1d0a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 --2- 172.21.15.153:0/361829820 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6dec0a2770 0x7f6dec1d0520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 --2- 172.21.15.153:0/361829820 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6dd803cea0 0x7f6dd803f360 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 --2- 172.21.15.153:0/361829820 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6dec06d5d0 0x7f6dec13fe80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:24.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 >> 172.21.15.153:0/361829820 conn(0x7f6dec0a7830 msgr2=0x7f6dec09d910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:24.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 shutdown_connections 2024-09-10T20:26:24.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 -- 172.21.15.153:0/361829820 wait complete. 2024-09-10T20:26:24.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.873+0000 7f6df10cd640 1 librados: shutdown 2024-09-10T20:26:24.894 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-10T20:26:23.069400+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163316,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067724,"statfs":{"total":579820584960,"available":579653349376,"internally_reserved":0,"allocated":1163264,"data_stored":1383940,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.723438"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773251+0000","last_change":"2024-09-10T20:26:21.773251+0000","last_active":"2024-09-10T20:26:21.773251+0000","last_peered":"2024-09-10T20:26:21.773251+0000","last_clean":"2024-09-10T20:26:21.773251+0000","last_became_active":"2024-09-10T20:26:19.356844+0000","last_became_peered":"2024-09-10T20:26:19.356844+0000","last_unstale":"2024-09-10T20:26:21.773251+0000","last_undegraded":"2024-09-10T20:26:21.773251+0000","last_fullsized":"2024-09-10T20:26:21.773251+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:23.413224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.381255+0000","last_change":"2024-09-10T20:26:21.381255+0000","last_active":"2024-09-10T20:26:21.381255+0000","last_peered":"2024-09-10T20:26:21.381255+0000","last_clean":"2024-09-10T20:26:21.381255+0000","last_became_active":"2024-09-10T20:26:19.363220+0000","last_became_peered":"2024-09-10T20:26:19.363220+0000","last_unstale":"2024-09-10T20:26:21.381255+0000","last_undegraded":"2024-09-10T20:26:21.381255+0000","last_fullsized":"2024-09-10T20:26:21.381255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:09.279291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096679600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773020+0000","last_change":"2024-09-10T20:26:21.773020+0000","last_active":"2024-09-10T20:26:21.773020+0000","last_peered":"2024-09-10T20:26:21.773020+0000","last_clean":"2024-09-10T20:26:21.773020+0000","last_became_active":"2024-09-10T20:26:19.363294+0000","last_became_peered":"2024-09-10T20:26:19.363294+0000","last_unstale":"2024-09-10T20:26:21.773020+0000","last_undegraded":"2024-09-10T20:26:21.773020+0000","last_fullsized":"2024-09-10T20:26:21.773020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:54.300619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053952999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.378377+0000","last_change":"2024-09-10T20:26:21.378377+0000","last_active":"2024-09-10T20:26:21.378377+0000","last_peered":"2024-09-10T20:26:21.378377+0000","last_clean":"2024-09-10T20:26:21.378377+0000","last_became_active":"2024-09-10T20:26:19.359961+0000","last_became_peered":"2024-09-10T20:26:19.359961+0000","last_unstale":"2024-09-10T20:26:21.378377+0000","last_undegraded":"2024-09-10T20:26:21.378377+0000","last_fullsized":"2024-09-10T20:26:21.378377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:21:01.940376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.394496+0000","last_change":"2024-09-10T20:26:21.386697+0000","last_active":"2024-09-10T20:26:21.394496+0000","last_peered":"2024-09-10T20:26:21.394496+0000","last_clean":"2024-09-10T20:26:21.394496+0000","last_became_active":"2024-09-10T20:26:19.360427+0000","last_became_peered":"2024-09-10T20:26:19.360427+0000","last_unstale":"2024-09-10T20:26:21.394496+0000","last_undegraded":"2024-09-10T20:26:21.394496+0000","last_fullsized":"2024-09-10T20:26:21.394496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:53.136006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020581079999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.575187+0000","last_change":"2024-09-10T20:26:21.575187+0000","last_active":"2024-09-10T20:26:21.575187+0000","last_peered":"2024-09-10T20:26:21.575187+0000","last_clean":"2024-09-10T20:26:21.575187+0000","last_became_active":"2024-09-10T20:26:19.362858+0000","last_became_peered":"2024-09-10T20:26:19.362858+0000","last_unstale":"2024-09-10T20:26:21.575187+0000","last_undegraded":"2024-09-10T20:26:21.575187+0000","last_fullsized":"2024-09-10T20:26:21.575187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:22:17.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078714099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.378915+0000","last_change":"2024-09-10T20:26:21.378915+0000","last_active":"2024-09-10T20:26:21.378915+0000","last_peered":"2024-09-10T20:26:21.378915+0000","last_clean":"2024-09-10T20:26:21.378915+0000","last_became_active":"2024-09-10T20:26:19.362158+0000","last_became_peered":"2024-09-10T20:26:19.362158+0000","last_unstale":"2024-09-10T20:26:21.378915+0000","last_undegraded":"2024-09-10T20:26:21.378915+0000","last_fullsized":"2024-09-10T20:26:21.378915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:50.708554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010122729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773025+0000","last_change":"2024-09-10T20:26:21.773025+0000","last_active":"2024-09-10T20:26:21.773025+0000","last_peered":"2024-09-10T20:26:21.773025+0000","last_clean":"2024-09-10T20:26:21.773025+0000","last_became_active":"2024-09-10T20:26:19.359775+0000","last_became_peered":"2024-09-10T20:26:19.359775+0000","last_unstale":"2024-09-10T20:26:21.773025+0000","last_undegraded":"2024-09-10T20:26:21.773025+0000","last_fullsized":"2024-09-10T20:26:21.773025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:35.427907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054594699999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.772391+0000","last_change":"2024-09-10T20:26:14.579551+0000","last_active":"2024-09-10T20:26:21.772391+0000","last_peered":"2024-09-10T20:26:21.772391+0000","last_clean":"2024-09-10T20:26:21.772391+0000","last_became_active":"2024-09-10T20:26:14.577602+0000","last_became_peered":"2024-09-10T20:26:14.577602+0000","last_unstale":"2024-09-10T20:26:21.772391+0000","last_undegraded":"2024-09-10T20:26:21.772391+0000","last_fullsized":"2024-09-10T20:26:21.772391+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_clean_scrub_stamp":"2024-09-10T20:26:13.287430+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:41.463707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574852,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574852,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":583269,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574851,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":583269,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":54341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:24.895 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:26:24.895 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-10T20:26:24.895 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-10T20:26:24.895 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-10T20:26:24.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient: get_monmap_and_config 2024-09-10T20:26:25.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient: build_initial_monmap 2024-09-10T20:26:25.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient: monmap: 2024-09-10T20:26:25.000 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:25.000 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.000 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:24.999251+0000 2024-09-10T20:26:25.000 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:24.999251+0000 2024-09-10T20:26:25.000 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:25.000 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.001 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:25.001 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:25.001 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:25.001 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:25.001 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:25.001 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 1 Processor -- start 2024-09-10T20:26:25.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 1 -- start start 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient: init 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient: _add_conns ranks=[0,3,2,4,1] 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f46641364f0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f46641360a0 0x7f466413b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient(hunting): picked mon.noname-a con 0x7f46641360a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46640a1990 0x7f466413d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient(hunting): picked mon.noname-b con 0x7f46640a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46640a24c0 con 0x7f46641364f0 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f46640a27a0 con 0x7f46641360a0 2024-09-10T20:26:25.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f46640a2a80 con 0x7f46640a1990 2024-09-10T20:26:25.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:25.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466b46f640 10 monclient(hunting): authenticate will time out at 945.138839s 2024-09-10T20:26:25.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f46641360a0 0x7f466413b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466920b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f46641364f0 0x7f46641368d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60522/0 (socket says 172.21.15.153:60522) 2024-09-10T20:26:25.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4669a0c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46640a1990 0x7f466413d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f466920b640 1 -- 172.21.15.153:0/3200875011 learned_addr learned my addr 172.21.15.153:0/3200875011 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:25.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient(hunting): get_auth_request con 0x7f46641360a0 auth_method 0 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4669a0c640 10 monclient(hunting): get_auth_request con 0x7f46640a1990 auth_method 0 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4669a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4669a0c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4669a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4669a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4669a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4669a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient(hunting): handle_auth_done global_id 4278 payload 274 2024-09-10T20:26:25.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 1 -- 172.21.15.153:0/3200875011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46640a1990 msgr2=0x7f466413d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:25.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 1 --2- 172.21.15.153:0/3200875011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46640a1990 0x7f466413d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 1 -- 172.21.15.153:0/3200875011 >> v1:172.21.15.153:6789/0 conn(0x7f46641364f0 legacy=0x7f46641368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:25.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient: _finish_hunting 0 2024-09-10T20:26:25.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 1 monclient: found mon.noname-a 2024-09-10T20:26:25.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:25.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4668a0a640 1 -- 172.21.15.153:0/3200875011 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46640a2d60 con 0x7f46641360a0 2024-09-10T20:26:25.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.993+0000 7f4669a0c640 1 --2- 172.21.15.153:0/3200875011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46640a1990 0x7f466413d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:25.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f4668a0a640 1 -- 172.21.15.153:0/3200875011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 msgr2=0x7f466413b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:25.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f4668a0a640 1 --2- 172.21.15.153:0/3200875011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 0x7f466413b420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f46600097c0 tx=0x7f466001c3f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde9b592bd7c299c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:25.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 1 -- 172.21.15.153:0/3200875011 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46600b2070 con 0x7f46641360a0 2024-09-10T20:26:25.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:25.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 10 monclient: dump: 2024-09-10T20:26:25.015 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.015 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 10 monclient: _finish_auth 0 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f466b46f640 5 monclient: authenticate success, global_id 4278 2024-09-10T20:26:25.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f466b46f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 1 -- 172.21.15.153:0/3200875011 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46600ad640 con 0x7f46641360a0 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f466b46f640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 1 -- 172.21.15.153:0/3200875011 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46600ada10 con 0x7f46641360a0 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f466b46f640 4 set_mon_vals no callback set 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:24.997+0000 7f465bfff640 10 monclient: dump: 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.017 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient: shutdown 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 -- 172.21.15.153:0/3200875011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 msgr2=0x7f466413b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 --2- 172.21.15.153:0/3200875011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 0x7f466413b420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f46600097c0 tx=0x7f466001c3f0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 -- 172.21.15.153:0/3200875011 shutdown_connections 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 --2- 172.21.15.153:0/3200875011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46640a1990 0x7f466413d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 --2- 172.21.15.153:0/3200875011 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 0x7f466413b420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 -- 172.21.15.153:0/3200875011 >> 172.21.15.153:0/3200875011 conn(0x7f46640a71b0 msgr2=0x7f46640a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 -- 172.21.15.153:0/3200875011 shutdown_connections 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 -- 172.21.15.153:0/3200875011 wait complete. 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient: build_initial_monmap 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient: monmap: 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:25.019 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 librados: starting msgr at 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 librados: starting objecter 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 Processor -- start 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 -- start start 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 librados: setting wanted keys 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 librados: calling monclient init 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient: init 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46640a1990 0x7f46641d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient(hunting): picked mon.noname-b con 0x7f46640a1990 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 0x7f46641d0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient(hunting): picked mon.noname-a con 0x7f46641360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46641364f0 0x7f46641d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient(hunting): picked mon.noname-c con 0x7f46641364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4664140ba0 con 0x7f46641360a0 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4664140a20 con 0x7f46640a1990 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4664140d20 con 0x7f46641364f0 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466b46f640 10 monclient(hunting): authenticate will time out at 945.145734s 2024-09-10T20:26:25.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46641364f0 0x7f46641d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 0x7f46641d0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 0x7f46641d0800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42342/0 (socket says 172.21.15.153:42342) 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46641364f0 0x7f46641d0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55772/0 (socket says 172.21.15.153:55772) 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 1 -- 172.21.15.153:0/937660736 learned_addr learned my addr 172.21.15.153:0/937660736 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient(hunting): get_auth_request con 0x7f46641360a0 auth_method 0 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 10 monclient(hunting): get_auth_request con 0x7f46641364f0 auth_method 0 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient(hunting): handle_auth_done global_id 4281 payload 824 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 1 -- 172.21.15.153:0/937660736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46641364f0 msgr2=0x7f46641d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 1 --2- 172.21.15.153:0/937660736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46641364f0 0x7f46641d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 1 -- 172.21.15.153:0/937660736 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46640a1990 msgr2=0x7f46641d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f466920b640 1 --2- 172.21.15.153:0/937660736 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46640a1990 0x7f46641d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 1 --2- 172.21.15.153:0/937660736 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46640a1990 0x7f46641d02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient: _finish_hunting 0 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 1 monclient: found mon.noname-a 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:25.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4668a0a640 1 -- 172.21.15.153:0/937660736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466413cb90 con 0x7f46641360a0 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.001+0000 7f4669a0c640 1 --2- 172.21.15.153:0/937660736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46641364f0 0x7f46641d0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4668a0a640 1 -- 172.21.15.153:0/937660736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 msgr2=0x7f46641d0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4668a0a640 1 --2- 172.21.15.153:0/937660736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 0x7f46641d0800 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f466001cc90 tx=0x7f4660019190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 client.?.objecter ms_handle_connect 0x7f46641360a0 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 1 -- 172.21.15.153:0/937660736 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46600ba070 con 0x7f46641360a0 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 monclient: dump: 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.024 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:25.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 1 -- 172.21.15.153:0/937660736 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46600b6820 con 0x7f46641360a0 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 5 monclient: authenticate success, global_id 4281 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 10 monclient: _renew_subs 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f466413bd00 con 0x7f46641360a0 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 1 -- 172.21.15.153:0/937660736 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46600b2040 con 0x7f46641360a0 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f465b7fe640 4 set_mon_vals no callback set 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 monclient: dump: 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.026 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 10 client.4281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 10 monclient: _renew_subs 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46640a6990 con 0x7f46641360a0 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 1 librados: init done 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 1 -- 172.21.15.153:0/937660736 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f46600bd070 con 0x7f46641360a0 2024-09-10T20:26:25.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46641368d0 con 0x7f46641360a0 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 1 --2- 172.21.15.153:0/937660736 >> v2:172.21.15.176:6812/3003473398 conn(0x7f464803cef0 0x7f464803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 1 -- 172.21.15.153:0/937660736 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f46600e92e0 con 0x7f46641360a0 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 10 client.4281.objecter ms_dispatch 0x7f4664000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 3 client.4281.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 3 client.4281.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f4659ffb640 20 client.4281.objecter dump_active .. 0 homeless 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466920b640 1 --2- 172.21.15.153:0/937660736 >> v2:172.21.15.176:6812/3003473398 conn(0x7f464803cef0 0x7f464803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.005+0000 7f466920b640 10 monclient: get_auth_request con 0x7f464803cef0 auth_method 0 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.009+0000 7f466920b640 1 --2- 172.21.15.153:0/937660736 >> v2:172.21.15.176:6812/3003473398 conn(0x7f464803cef0 0x7f464803f3b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4654005060 tx=0x7f4654009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:25.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.009+0000 7f4659ffb640 10 client.4281.objecter ms_handle_connect 0x7f464803cef0 2024-09-10T20:26:25.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.009+0000 7f4659ffb640 1 -- 172.21.15.153:0/937660736 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f4660016070 con 0x7f46641360a0 2024-09-10T20:26:25.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.009+0000 7f4659ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:25.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.009+0000 7f4659ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:25.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-10T20:26:25.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-10T20:26:25.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f466406a640 con 0x7f46641360a0 2024-09-10T20:26:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f4659ffb640 1 -- 172.21.15.153:0/937660736 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f46600e81a0 con 0x7f46641360a0 2024-09-10T20:26:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f4659ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-10T20:26:25.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f4659ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:25.220 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:25.220 INFO:teuthology.orchestra.run.smithi153.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-10T20:26:25.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 10 librados: watch_flush enter 2024-09-10T20:26:25.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 10 librados: watch_flush exit 2024-09-10T20:26:25.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 20 client.4281.objecter shutdown clearing up homeless session... 2024-09-10T20:26:25.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 10 client.4281.objecter successfully canceled tick 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 >> v2:172.21.15.176:6812/3003473398 conn(0x7f464803cef0 msgr2=0x7f464803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 --2- 172.21.15.153:0/937660736 >> v2:172.21.15.176:6812/3003473398 conn(0x7f464803cef0 0x7f464803f3b0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4654005060 tx=0x7f4654009040 comp rx=0 tx=0).stop 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 10 monclient: shutdown 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 msgr2=0x7f46641d0800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 --2- 172.21.15.153:0/937660736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 0x7f46641d0800 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f466001cc90 tx=0x7f4660019190 comp rx=0 tx=0).stop 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 shutdown_connections 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 --2- 172.21.15.153:0/937660736 >> v2:172.21.15.176:6812/3003473398 conn(0x7f464803cef0 0x7f464803f3b0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 --2- 172.21.15.153:0/937660736 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f46641364f0 0x7f46641d0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 --2- 172.21.15.153:0/937660736 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f46641360a0 0x7f46641d0800 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 --2- 172.21.15.153:0/937660736 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f46640a1990 0x7f46641d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 >> 172.21.15.153:0/937660736 conn(0x7f46640a71b0 msgr2=0x7f46640a57f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 shutdown_connections 2024-09-10T20:26:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 -- 172.21.15.153:0/937660736 wait complete. 2024-09-10T20:26:25.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.213+0000 7f466b46f640 1 librados: shutdown 2024-09-10T20:26:25.237 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-10T20:26:25.238 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-10T20:26:25.256 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-10T20:26:25.256 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:26:25.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient: get_monmap_and_config 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient: build_initial_monmap 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient: monmap: 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:25.367232+0000 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:25.367232+0000 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:25.368 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:25.369 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:25.369 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 1 Processor -- start 2024-09-10T20:26:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 1 -- start start 2024-09-10T20:26:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient: init 2024-09-10T20:26:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:25.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa6280a5bb0 0x7fa6280a5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient(hunting): picked mon.noname-a con 0x7fa6280a5bb0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa62809dd10 0x7fa6280a64d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient(hunting): picked mon.noname-c con 0x7fa62809dd10 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa628138970 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6280707f0 con 0x7fa628138970 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fa62809d3e0 con 0x7fa6280a5bb0 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa6280a6a10 con 0x7fa62809dd10 2024-09-10T20:26:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62f419640 10 monclient(hunting): authenticate will time out at 945.507668s 2024-09-10T20:26:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62d1b5640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa6280a5bb0 0x7fa6280a5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62d1b5640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fa6280a5bb0 0x7fa6280a5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42346/0 (socket says 172.21.15.153:42346) 2024-09-10T20:26:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62d1b5640 1 -- 172.21.15.153:0/485162301 learned_addr learned my addr 172.21.15.153:0/485162301 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62d1b5640 10 monclient(hunting): get_auth_request con 0x7fa6280a5bb0 auth_method 0 2024-09-10T20:26:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62d1b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62d1b5640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.361+0000 7fa62d1b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62c9b4640 1 --2- 172.21.15.153:0/485162301 >> v2:172.21.15.176:3300/0 conn(0x7fa62809dd10 0x7fa6280a64d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62c9b4640 10 monclient(hunting): get_auth_request con 0x7fa62809dd10 auth_method 0 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62c9b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62c9b4640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62c9b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 10 monclient(hunting): handle_auth_done global_id 4284 payload 274 2024-09-10T20:26:25.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 1 -- 172.21.15.153:0/485162301 >> v2:172.21.15.176:3300/0 conn(0x7fa62809dd10 msgr2=0x7fa6280a64d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 1 --2- 172.21.15.153:0/485162301 >> v2:172.21.15.176:3300/0 conn(0x7fa62809dd10 0x7fa6280a64d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 1 -- 172.21.15.153:0/485162301 >> v1:172.21.15.176:6789/0 conn(0x7fa628138970 legacy=0x7fa628138dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 10 monclient: _finish_hunting 0 2024-09-10T20:26:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 1 monclient: found mon.noname-a 2024-09-10T20:26:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.365+0000 7fa62d1b5640 1 -- 172.21.15.153:0/485162301 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa628070a50 con 0x7fa6280a5bb0 2024-09-10T20:26:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62d1b5640 1 -- 172.21.15.153:0/485162301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6280a5bb0 msgr2=0x7fa6280a5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62d1b5640 1 --2- 172.21.15.153:0/485162301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6280a5bb0 0x7fa6280a5f90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa6280a7c40 tx=0x7fa61801c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b35b10cb720ed73 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 1 -- 172.21.15.153:0/485162301 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6180b0070 con 0x7fa6280a5bb0 2024-09-10T20:26:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:25.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 10 monclient: dump: 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 10 monclient: _finish_auth 0 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 1 -- 172.21.15.153:0/485162301 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6180ab600 con 0x7fa6280a5bb0 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 5 monclient: authenticate success, global_id 4284 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 1 -- 172.21.15.153:0/485162301 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6180ab9d0 con 0x7fa6280a5bb0 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 4 set_mon_vals no callback set 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa61ffff640 10 monclient: dump: 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.378 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.379 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.379 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient: shutdown 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 -- 172.21.15.153:0/485162301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6280a5bb0 msgr2=0x7fa6280a5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 --2- 172.21.15.153:0/485162301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6280a5bb0 0x7fa6280a5f90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa6280a7c40 tx=0x7fa61801c3b0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 -- 172.21.15.153:0/485162301 shutdown_connections 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 --2- 172.21.15.153:0/485162301 >> v2:172.21.15.176:3300/0 conn(0x7fa62809dd10 0x7fa6280a64d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 --2- 172.21.15.153:0/485162301 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa6280a5bb0 0x7fa6280a5f90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 -- 172.21.15.153:0/485162301 >> 172.21.15.153:0/485162301 conn(0x7fa6280a7cb0 msgr2=0x7fa6280a80c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 -- 172.21.15.153:0/485162301 shutdown_connections 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 -- 172.21.15.153:0/485162301 wait complete. 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient: build_initial_monmap 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient: monmap: 2024-09-10T20:26:25.385 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 librados: starting msgr at 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 librados: starting objecter 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 Processor -- start 2024-09-10T20:26:25.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 -- start start 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 librados: setting wanted keys 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 librados: calling monclient init 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient: init 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa628138970 0x7fa6281d09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient(hunting): picked mon.noname-b con 0x7fa628138970 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa628137af0 0x7fa62813da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient(hunting): picked mon.noname-c con 0x7fa628137af0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:25.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa628137f70 0x7fa62813df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient(hunting): picked mon.noname-a con 0x7fa628137f70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa6281435e0 con 0x7fa628137f70 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa628143460 con 0x7fa628138970 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa628143760 con 0x7fa628137af0 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.369+0000 7fa62f419640 10 monclient(hunting): authenticate will time out at 945.515967s 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa628137f70 0x7fa62813df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62c9b4640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa628137af0 0x7fa62813da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa628137f70 0x7fa62813df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42356/0 (socket says 172.21.15.153:42356) 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 -- 172.21.15.153:0/3422353160 learned_addr learned my addr 172.21.15.153:0/3422353160 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient(hunting): get_auth_request con 0x7fa628137f70 auth_method 0 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62c9b4640 10 monclient(hunting): get_auth_request con 0x7fa628137af0 auth_method 0 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62c9b4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62c9b4640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62c9b4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62c9b4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62c9b4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62c9b4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient(hunting): handle_auth_done global_id 4287 payload 824 2024-09-10T20:26:25.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d1b5640 1 --2- 172.21.15.153:0/3422353160 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa628138970 0x7fa6281d09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 -- 172.21.15.153:0/3422353160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa628137af0 msgr2=0x7fa62813da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 --2- 172.21.15.153:0/3422353160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa628137af0 0x7fa62813da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 -- 172.21.15.153:0/3422353160 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa628138970 msgr2=0x7fa6281d09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 --2- 172.21.15.153:0/3422353160 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa628138970 0x7fa6281d09c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient: _finish_hunting 0 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 monclient: found mon.noname-a 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 -- 172.21.15.153:0/3422353160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6281221a0 con 0x7fa628137f70 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 -- 172.21.15.153:0/3422353160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa628137f70 msgr2=0x7fa62813df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62d9b6640 1 --2- 172.21.15.153:0/3422353160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa628137f70 0x7fa62813df80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa62000c7b0 tx=0x7fa62000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 client.?.objecter ms_handle_connect 0x7fa628137f70 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 1 -- 172.21.15.153:0/3422353160 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa62000ac60 con 0x7fa628137f70 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 monclient: dump: 2024-09-10T20:26:25.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.394 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.394 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.394 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.394 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.394 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 5 monclient: authenticate success, global_id 4287 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 1 -- 172.21.15.153:0/3422353160 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa62000ae00 con 0x7fa628137f70 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 10 monclient: _renew_subs 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6280a4fb0 con 0x7fa628137f70 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 1 -- 172.21.15.153:0/3422353160 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa620004bb0 con 0x7fa628137f70 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61effd640 4 set_mon_vals no callback set 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 10 monclient: dump: 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:25.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 10 monclient: _renew_subs 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa628121800 con 0x7fa628137f70 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 1 librados: init done 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa628138dc0 con 0x7fa628137f70 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.373+0000 7fa61dffb640 1 -- 172.21.15.153:0/3422353160 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fa620005030 con 0x7fa628137f70 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa61dffb640 1 --2- 172.21.15.153:0/3422353160 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa60c03ced0 0x7fa60c03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa61dffb640 1 -- 172.21.15.153:0/3422353160 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7fa62004f4f0 con 0x7fa628137f70 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa61dffb640 10 client.4287.objecter ms_dispatch 0x7fa628000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:25.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa61dffb640 3 client.4287.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:25.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa61dffb640 3 client.4287.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:25.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa61dffb640 20 client.4287.objecter dump_active .. 0 homeless 2024-09-10T20:26:25.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa62d1b5640 1 --2- 172.21.15.153:0/3422353160 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa60c03ced0 0x7fa60c03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa62d1b5640 10 monclient: get_auth_request con 0x7fa60c03ced0 auth_method 0 2024-09-10T20:26:25.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa61dffb640 10 client.4287.objecter ms_handle_connect 0x7fa60c03ced0 2024-09-10T20:26:25.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.377+0000 7fa62d1b5640 1 --2- 172.21.15.153:0/3422353160 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa60c03ced0 0x7fa60c03f390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa62809d7e0 tx=0x7fa618008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:25.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.381+0000 7fa61dffb640 1 -- 172.21.15.153:0/3422353160 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fa620020070 con 0x7fa628137f70 2024-09-10T20:26:25.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.381+0000 7fa61dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:25.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.381+0000 7fa61dffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:25.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.545+0000 7fa62f419640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:25.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.545+0000 7fa62f419640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:25.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.545+0000 7fa62f419640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.545+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6280a6c70 con 0x7fa628137f70 2024-09-10T20:26:25.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.545+0000 7fa61dffb640 1 -- 172.21.15.153:0/3422353160 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10987 (secure 0 0 0) 0x7fa620005320 con 0x7fa628137f70 2024-09-10T20:26:25.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.545+0000 7fa61dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:26:25.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.545+0000 7fa61dffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:25.554 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:25.554 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":20,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","created":"2024-09-10T20:25:59.587062+0000","modified":"2024-09-10T20:26:21.366705+0000","last_up_change":"2024-09-10T20:26:12.282988+0000","last_in_change":"2024-09-10T20:26:02.414479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:26:13.125072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-10T20:26:18.322750+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"faf817a9-b230-4970-b8c5-e5637fd29193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4216589932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":4216589932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":4216589932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4216589932}]},"public_addr":"172.21.15.153:6800/4216589932","cluster_addr":"172.21.15.153:6801/4216589932","heartbeat_back_addr":"172.21.15.153:6803/4216589932","heartbeat_front_addr":"172.21.15.153:6802/4216589932","state":["exists","up"]},{"osd":1,"uuid":"f85962f8-8bd9-4e26-8d66-49b6528617a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1715391555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":1715391555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1715391555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1715391555}]},"public_addr":"172.21.15.153:6804/1715391555","cluster_addr":"172.21.15.153:6805/1715391555","heartbeat_back_addr":"172.21.15.153:6807/1715391555","heartbeat_front_addr":"172.21.15.153:6806/1715391555","state":["exists","up"]},{"osd":2,"uuid":"18d54b55-1393-42b9-9fcc-84202659cf31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1626005750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":1626005750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":1626005750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1626005750}]},"public_addr":"172.21.15.153:6808/1626005750","cluster_addr":"172.21.15.153:6809/1626005750","heartbeat_back_addr":"172.21.15.153:6811/1626005750","heartbeat_front_addr":"172.21.15.153:6810/1626005750","state":["exists","up"]},{"osd":3,"uuid":"5149b9c4-35d7-4ffb-9fac-7b3d9b660101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2953083416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":2953083416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6811","nonce":2953083416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2953083416}]},"public_addr":"172.21.15.176:6808/2953083416","cluster_addr":"172.21.15.176:6809/2953083416","heartbeat_back_addr":"172.21.15.176:6811/2953083416","heartbeat_front_addr":"172.21.15.176:6810/2953083416","state":["exists","up"]},{"osd":4,"uuid":"61e0c0cc-4301-4947-9f1d-838d2d7bbfea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4030285519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6805","nonce":4030285519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6807","nonce":4030285519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4030285519}]},"public_addr":"172.21.15.176:6804/4030285519","cluster_addr":"172.21.15.176:6805/4030285519","heartbeat_back_addr":"172.21.15.176:6807/4030285519","heartbeat_front_addr":"172.21.15.176:6806/4030285519","state":["exists","up"]},{"osd":5,"uuid":"2feacd82-c45a-4d90-8210-ff9faecc2d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":2372911183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6801","nonce":2372911183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6803","nonce":2372911183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2372911183}]},"public_addr":"172.21.15.176:6800/2372911183","cluster_addr":"172.21.15.176:6801/2372911183","heartbeat_back_addr":"172.21.15.176:6803/2372911183","heartbeat_front_addr":"172.21.15.176:6802/2372911183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.301670+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:10.037323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.733818+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364748+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.173071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:26:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 10 librados: watch_flush enter 2024-09-10T20:26:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 10 librados: watch_flush exit 2024-09-10T20:26:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 20 client.4287.objecter shutdown clearing up homeless session... 2024-09-10T20:26:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 10 client.4287.objecter successfully canceled tick 2024-09-10T20:26:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa60c03ced0 msgr2=0x7fa60c03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 --2- 172.21.15.153:0/3422353160 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa60c03ced0 0x7fa60c03f390 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa62809d7e0 tx=0x7fa618008040 comp rx=0 tx=0).stop 2024-09-10T20:26:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 10 monclient: shutdown 2024-09-10T20:26:25.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa628137f70 msgr2=0x7fa62813df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 --2- 172.21.15.153:0/3422353160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa628137f70 0x7fa62813df80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa62000c7b0 tx=0x7fa62000cc70 comp rx=0 tx=0).stop 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 shutdown_connections 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 --2- 172.21.15.153:0/3422353160 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa628137f70 0x7fa62813df80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 --2- 172.21.15.153:0/3422353160 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa60c03ced0 0x7fa60c03f390 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 --2- 172.21.15.153:0/3422353160 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa628137af0 0x7fa62813da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 --2- 172.21.15.153:0/3422353160 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa628138970 0x7fa6281d09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 >> 172.21.15.153:0/3422353160 conn(0x7fa6280a7cb0 msgr2=0x7fa6280a90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 shutdown_connections 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 -- 172.21.15.153:0/3422353160 wait complete. 2024-09-10T20:26:25.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.549+0000 7fa62f419640 1 librados: shutdown 2024-09-10T20:26:25.571 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-10T20:26:13.125072+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-10T20:26:18.322750+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-10T20:26:25.572 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-10T20:26:25.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.673+0000 7f1c937c7640 10 monclient: get_monmap_and_config 2024-09-10T20:26:25.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.673+0000 7f1c937c7640 10 monclient: build_initial_monmap 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.673+0000 7f1c937c7640 10 monclient: monmap: 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:25.681184+0000 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:25.681184+0000 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:25.682 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 1 Processor -- start 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 1 -- start start 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 10 monclient: init 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c09e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 10 monclient(hunting): picked mon.noname-b con 0x7f1c8c09e1a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1c8c0a27b0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1c8c13a6b0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c8c0a3410 con 0x7f1c8c0a27b0 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c8c0a10e0 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8c0a2ca0 con 0x7f1c8c09e1a0 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:25.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c937c7640 10 monclient(hunting): authenticate will time out at 945.820775s 2024-09-10T20:26:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c90d62640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1c8c0a27b0 0x7f1c8c09eac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60536/0 (socket says 172.21.15.153:60536) 2024-09-10T20:26:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c09e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c90d62640 1 -- 172.21.15.153:0/62353952 learned_addr learned my addr 172.21.15.153:0/62353952 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient(hunting): get_auth_request con 0x7f1c8c09e1a0 auth_method 0 2024-09-10T20:26:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient(hunting): handle_auth_done global_id 4295 payload 274 2024-09-10T20:26:25.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 1 -- 172.21.15.153:0/62353952 >> v1:172.21.15.176:6789/0 conn(0x7f1c8c13a6b0 legacy=0x7f1c8c13caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:25.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 1 -- 172.21.15.153:0/62353952 >> v1:172.21.15.153:6789/0 conn(0x7f1c8c0a27b0 legacy=0x7f1c8c09eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:25.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient: _finish_hunting 0 2024-09-10T20:26:25.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 1 monclient: found mon.noname-b 2024-09-10T20:26:25.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:25.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:25.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.677+0000 7f1c91563640 1 -- 172.21.15.153:0/62353952 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c09fde0 con 0x7f1c8c09e1a0 2024-09-10T20:26:25.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c91563640 1 -- 172.21.15.153:0/62353952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 msgr2=0x7f1c8c09e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:25.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c91563640 1 --2- 172.21.15.153:0/62353952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c09e580 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0097c0 tx=0x7f1c7c01c3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21cc8092eae8c7d0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 1 -- 172.21.15.153:0/62353952 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0b0070 con 0x7f1c8c09e1a0 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 10 monclient: dump: 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.690 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 10 monclient: _finish_auth 0 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 1 -- 172.21.15.153:0/62353952 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c7c0ab600 con 0x7f1c8c09e1a0 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 1 -- 172.21.15.153:0/62353952 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c7c0ab9d0 con 0x7f1c8c09e1a0 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 5 monclient: authenticate success, global_id 4295 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 4 set_mon_vals no callback set 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c83fff640 10 monclient: dump: 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.691 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 10 monclient: shutdown 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 -- 172.21.15.153:0/62353952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 msgr2=0x7f1c8c09e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 --2- 172.21.15.153:0/62353952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c09e580 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0097c0 tx=0x7f1c7c01c3f0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 -- 172.21.15.153:0/62353952 shutdown_connections 2024-09-10T20:26:25.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 --2- 172.21.15.153:0/62353952 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c09e580 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 -- 172.21.15.153:0/62353952 >> 172.21.15.153:0/62353952 conn(0x7f1c8c0a7890 msgr2=0x7f1c8c0a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 -- 172.21.15.153:0/62353952 shutdown_connections 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 -- 172.21.15.153:0/62353952 wait complete. 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 10 monclient: build_initial_monmap 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 10 monclient: monmap: 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:25.693 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 librados: starting msgr at 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 librados: starting objecter 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 Processor -- start 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 -- start start 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 librados: setting wanted keys 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 librados: calling monclient init 2024-09-10T20:26:25.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 10 monclient: init 2024-09-10T20:26:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c11ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 10 monclient(hunting): picked mon.noname-c con 0x7f1c8c09e1a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c0a27b0 0x7f1c8c135770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 10 monclient(hunting): picked mon.noname-b con 0x7f1c8c0a27b0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:25.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.681+0000 7f1c937c7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c13a6b0 0x7f1c8c11d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient(hunting): picked mon.noname-a con 0x7f1c8c13a6b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c140dc0 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1c8c140c40 con 0x7f1c8c0a27b0 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1c8c140f40 con 0x7f1c8c09e1a0 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient(hunting): authenticate will time out at 945.828051s 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c11ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c11ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55810/0 (socket says 172.21.15.153:55810) 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 1 -- 172.21.15.153:0/3129875189 learned_addr learned my addr 172.21.15.153:0/3129875189 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 10 monclient(hunting): get_auth_request con 0x7f1c8c09e1a0 auth_method 0 2024-09-10T20:26:25.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c13a6b0 0x7f1c8c11d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient(hunting): get_auth_request con 0x7f1c8c13a6b0 auth_method 0 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:25.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91563640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 1 -- 172.21.15.153:0/3129875189 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 msgr2=0x7f1c8c11ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c90d62640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c0a27b0 0x7f1c8c135770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c11ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 1 -- 172.21.15.153:0/3129875189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c0a27b0 msgr2=0x7f1c8c135770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c0a27b0 0x7f1c8c135770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient: _finish_hunting 0 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 1 monclient: found mon.noname-a 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 1 -- 172.21.15.153:0/3129875189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c8c073f00 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c90d62640 10 monclient: get_auth_request con 0x7f1c8c0a27b0 auth_method 0 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c90d62640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c0a27b0 0x7f1c8c135770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:26:25.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 1 -- 172.21.15.153:0/3129875189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c13a6b0 msgr2=0x7f1c8c11d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c91d64640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c13a6b0 0x7f1c8c11d2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400c780 tx=0x7f1c8400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 client.?.objecter ms_handle_connect 0x7f1c8c13a6b0 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 1 -- 172.21.15.153:0/3129875189 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84016020 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 monclient: dump: 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.703 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 monclient: _finish_auth 0 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 1 -- 172.21.15.153:0/3129875189 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8400ac60 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 1 -- 172.21.15.153:0/3129875189 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84012660 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 5 monclient: authenticate success, global_id 4290 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient: _renew_subs 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c8c075220 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c837fe640 4 set_mon_vals no callback set 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 10 monclient: dump: 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:25.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient: _renew_subs 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c8c120200 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 1 librados: init done 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c8c09e580 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.685+0000 7f1c81ffb640 1 -- 172.21.15.153:0/3129875189 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f1c84019020 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c81ffb640 1 --2- 172.21.15.153:0/3129875189 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1c7003cea0 0x7f1c7003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c81ffb640 1 -- 172.21.15.153:0/3129875189 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f1c8404f160 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c81ffb640 10 client.4290.objecter ms_dispatch 0x7f1c8c000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:25.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c81ffb640 3 client.4290.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:25.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c81ffb640 3 client.4290.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:25.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c81ffb640 20 client.4290.objecter dump_active .. 0 homeless 2024-09-10T20:26:25.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c91563640 1 --2- 172.21.15.153:0/3129875189 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1c7003cea0 0x7f1c7003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:25.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c91563640 10 monclient: get_auth_request con 0x7f1c7003cea0 auth_method 0 2024-09-10T20:26:25.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c91563640 1 --2- 172.21.15.153:0/3129875189 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1c7003cea0 0x7f1c7003f360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c01c430 tx=0x7f1c7c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:25.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.689+0000 7f1c81ffb640 10 client.4290.objecter ms_handle_connect 0x7f1c7003cea0 2024-09-10T20:26:25.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.693+0000 7f1c81ffb640 1 -- 172.21.15.153:0/3129875189 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f1c84020030 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.693+0000 7f1c81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:25.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.693+0000 7f1c81ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:25.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-10T20:26:25.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-10T20:26:25.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:25.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1c8c11fa50 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c81ffb640 1 -- 172.21.15.153:0/3129875189 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f1c8c075220 con 0x7f1c8c13a6b0 2024-09-10T20:26:25.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-10T20:26:25.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c81ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:25.864 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 1 2024-09-10T20:26:25.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 10 librados: watch_flush enter 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 10 librados: watch_flush exit 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 20 client.4290.objecter shutdown clearing up homeless session... 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 10 client.4290.objecter successfully canceled tick 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1c7003cea0 msgr2=0x7f1c7003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 --2- 172.21.15.153:0/3129875189 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1c7003cea0 0x7f1c7003f360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c01c430 tx=0x7f1c7c008040 comp rx=0 tx=0).stop 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 10 monclient: shutdown 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c13a6b0 msgr2=0x7f1c8c11d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c13a6b0 0x7f1c8c11d2f0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400c780 tx=0x7f1c8400cc40 comp rx=0 tx=0).stop 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 shutdown_connections 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1c8c13a6b0 0x7f1c8c11d2f0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1c8c0a27b0 0x7f1c8c135770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 --2- 172.21.15.153:0/3129875189 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1c7003cea0 0x7f1c7003f360 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.857+0000 7f1c937c7640 1 --2- 172.21.15.153:0/3129875189 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1c8c09e1a0 0x7f1c8c11ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:25.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.861+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 >> 172.21.15.153:0/3129875189 conn(0x7f1c8c0a7890 msgr2=0x7f1c8c0a15a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:25.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.861+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 shutdown_connections 2024-09-10T20:26:25.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.861+0000 7f1c937c7640 1 -- 172.21.15.153:0/3129875189 wait complete. 2024-09-10T20:26:25.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.861+0000 7f1c937c7640 1 librados: shutdown 2024-09-10T20:26:25.882 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-10T20:26:26.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.997+0000 7f12bea02640 10 monclient: get_monmap_and_config 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.997+0000 7f12bea02640 10 monclient: build_initial_monmap 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.997+0000 7f12bea02640 10 monclient: monmap: 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:26.004296+0000 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:26.004296+0000 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:26.005 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:26.006 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:26.006 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:26.006 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.997+0000 7f12bea02640 1 Processor -- start 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.997+0000 7f12bea02640 1 -- start start 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:25.997+0000 7f12bea02640 10 monclient: init 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b80a3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 10 monclient(hunting): picked mon.noname-b con 0x7f12b80a5a40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f12b80a5e20 0x7f12b80a6280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 10 monclient(hunting): picked mon.noname-a con 0x7f12b80a5e20 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f12b81388e0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12b80a6b70 con 0x7f12b81388e0 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f12b80708a0 con 0x7f12b80a5e20 2024-09-10T20:26:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f12b80a26b0 con 0x7f12b80a5a40 2024-09-10T20:26:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12bea02640 10 monclient(hunting): authenticate will time out at 946.144419s 2024-09-10T20:26:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b80a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b77fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f12b80a5e20 0x7f12b80a6280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b80a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55818/0 (socket says 172.21.15.153:55818) 2024-09-10T20:26:26.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 1 -- 172.21.15.153:0/3671304008 learned_addr learned my addr 172.21.15.153:0/3671304008 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient(hunting): get_auth_request con 0x7f12b80a5a40 auth_method 0 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b77fe640 10 monclient(hunting): get_auth_request con 0x7f12b80a5e20 auth_method 0 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient(hunting): handle_auth_done global_id 4301 payload 274 2024-09-10T20:26:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 1 -- 172.21.15.153:0/3671304008 >> v2:172.21.15.153:3300/0 conn(0x7f12b80a5e20 msgr2=0x7f12b80a6280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 1 --2- 172.21.15.153:0/3671304008 >> v2:172.21.15.153:3300/0 conn(0x7f12b80a5e20 0x7f12b80a6280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 1 -- 172.21.15.153:0/3671304008 >> v1:172.21.15.153:6789/0 conn(0x7f12b81388e0 legacy=0x7f12b8138d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:26.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 1 monclient: found mon.noname-b 2024-09-10T20:26:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:26.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.001+0000 7f12b7fff640 1 -- 172.21.15.153:0/3671304008 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b80705f0 con 0x7f12b80a5a40 2024-09-10T20:26:26.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 -- 172.21.15.153:0/3671304008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 msgr2=0x7f12b80a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:26.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 --2- 172.21.15.153:0/3671304008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b80a3930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009e50 tx=0x7f12a801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5af03b833819673 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 1 -- 172.21.15.153:0/3671304008 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a80a5530 con 0x7f12b80a5a40 2024-09-10T20:26:26.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 10 monclient: dump: 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:26.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 1 -- 172.21.15.153:0/3671304008 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a80a56d0 con 0x7f12b80a5a40 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 5 monclient: authenticate success, global_id 4301 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 1 -- 172.21.15.153:0/3671304008 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a80a5aa0 con 0x7f12b80a5a40 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 4 set_mon_vals no callback set 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b6ffd640 10 monclient: dump: 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.017 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient: shutdown 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 -- 172.21.15.153:0/3671304008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 msgr2=0x7f12b80a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 --2- 172.21.15.153:0/3671304008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b80a3930 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f12a8009e50 tx=0x7f12a801c8f0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 -- 172.21.15.153:0/3671304008 shutdown_connections 2024-09-10T20:26:26.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 --2- 172.21.15.153:0/3671304008 >> v2:172.21.15.153:3300/0 conn(0x7f12b80a5e20 0x7f12b80a6280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 --2- 172.21.15.153:0/3671304008 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b80a3930 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 -- 172.21.15.153:0/3671304008 >> 172.21.15.153:0/3671304008 conn(0x7f12b80ab980 msgr2=0x7f12b80a7b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 -- 172.21.15.153:0/3671304008 shutdown_connections 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 -- 172.21.15.153:0/3671304008 wait complete. 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient: build_initial_monmap 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient: monmap: 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:26.019 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 librados: starting msgr at 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 librados: starting objecter 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 Processor -- start 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 -- start start 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 librados: setting wanted keys 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 librados: calling monclient init 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient: init 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:26.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b81376f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient(hunting): picked mon.noname-c con 0x7f12b80a5a40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12b80a5e20 0x7f12b813d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient(hunting): picked mon.noname-b con 0x7f12b80a5e20 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b81388e0 0x7f12b813dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient(hunting): picked mon.noname-a con 0x7f12b81388e0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12b8143760 con 0x7f12b81388e0 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f12b81435e0 con 0x7f12b80a5e20 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f12b81438e0 con 0x7f12b80a5a40 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:26.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bea02640 10 monclient(hunting): authenticate will time out at 946.151346s 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b81376f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b81376f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55834/0 (socket says 172.21.15.153:55834) 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 -- 172.21.15.153:0/2855178738 learned_addr learned my addr 172.21.15.153:0/2855178738 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient(hunting): get_auth_request con 0x7f12b80a5a40 auth_method 0 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bcf9f640 1 --2- 172.21.15.153:0/2855178738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b81388e0 0x7f12b813dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bcf9f640 10 monclient(hunting): get_auth_request con 0x7f12b81388e0 auth_method 0 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bcf9f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bcf9f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bcf9f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bcf9f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bcf9f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12bcf9f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient(hunting): handle_auth_done global_id 4304 payload 824 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 -- 172.21.15.153:0/2855178738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12b80a5e20 msgr2=0x7f12b813d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 --2- 172.21.15.153:0/2855178738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12b80a5e20 0x7f12b813d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 -- 172.21.15.153:0/2855178738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b81388e0 msgr2=0x7f12b813dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 --2- 172.21.15.153:0/2855178738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b81388e0 0x7f12b813dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 monclient: found mon.noname-c 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:26.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.005+0000 7f12b7fff640 1 -- 172.21.15.153:0/2855178738 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12b8122160 con 0x7f12b80a5a40 2024-09-10T20:26:26.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b7fff640 1 -- 172.21.15.153:0/2855178738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 msgr2=0x7f12b81376f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:26.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b7fff640 1 --2- 172.21.15.153:0/2855178738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b81376f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12a801d190 tx=0x7f12a8019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 client.?.objecter ms_handle_connect 0x7f12b80a5a40 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 1 -- 172.21.15.153:0/2855178738 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a80bc020 con 0x7f12b80a5a40 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 monclient: dump: 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.026 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 monclient: _finish_auth 0 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 monclient: _check_auth_tickets 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 1 -- 172.21.15.153:0/2855178738 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12a80b8880 con 0x7f12b80a5a40 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 5 monclient: authenticate success, global_id 4304 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 10 monclient: _renew_subs 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12b813f320 con 0x7f12b80a5a40 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 1 -- 172.21.15.153:0/2855178738 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a80a5530 con 0x7f12b80a5a40 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:26.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 monclient: dump: 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b67fc640 4 set_mon_vals no callback set 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 10 monclient: _renew_subs 2024-09-10T20:26:26.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f12b8140150 con 0x7f12b80a5a40 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 1 librados: init done 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12b80a3930 con 0x7f12b80a5a40 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 1 -- 172.21.15.153:0/2855178738 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f12a80bf020 con 0x7f12b80a5a40 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 1 --2- 172.21.15.153:0/2855178738 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12a403cef0 0x7f12a403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 1 -- 172.21.15.153:0/2855178738 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f12a80ea110 con 0x7f12b80a5a40 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 10 client.4304.objecter ms_dispatch 0x7f12b8000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 3 client.4304.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 3 client.4304.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b4ff9640 20 client.4304.objecter dump_active .. 0 homeless 2024-09-10T20:26:26.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b77fe640 1 --2- 172.21.15.153:0/2855178738 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12a403cef0 0x7f12a403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.009+0000 7f12b77fe640 10 monclient: get_auth_request con 0x7f12a403cef0 auth_method 0 2024-09-10T20:26:26.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.013+0000 7f12b4ff9640 10 client.4304.objecter ms_handle_connect 0x7f12a403cef0 2024-09-10T20:26:26.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.013+0000 7f12b77fe640 1 --2- 172.21.15.153:0/2855178738 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12a403cef0 0x7f12a403f3b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12ac006aa0 tx=0x7f12ac006a30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.013+0000 7f12b4ff9640 1 -- 172.21.15.153:0/2855178738 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f12a80c6030 con 0x7f12b80a5a40 2024-09-10T20:26:26.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.013+0000 7f12b4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.013+0000 7f12b4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:26.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.181+0000 7f12bea02640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-10T20:26:26.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.181+0000 7f12bea02640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-10T20:26:26.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.181+0000 7f12bea02640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:26.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.181+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f12b80a0f30 con 0x7f12b80a5a40 2024-09-10T20:26:26.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.181+0000 7f12b4ff9640 1 -- 172.21.15.153:0/2855178738 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (secure 0 0 0) 0x7f12a8016600 con 0x7f12b80a5a40 2024-09-10T20:26:26.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12b4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-10T20:26:26.190 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12b4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:26.190 INFO:teuthology.orchestra.run.smithi153.stdout:pg_num: 8 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 10 librados: watch_flush enter 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 10 librados: watch_flush exit 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 20 client.4304.objecter shutdown clearing up homeless session... 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 10 client.4304.objecter successfully canceled tick 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12a403cef0 msgr2=0x7f12a403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 --2- 172.21.15.153:0/2855178738 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12a403cef0 0x7f12a403f3b0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f12ac006aa0 tx=0x7f12ac006a30 comp rx=0 tx=0).stop 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 10 monclient: shutdown 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 msgr2=0x7f12b81376f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 --2- 172.21.15.153:0/2855178738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b81376f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f12a801d190 tx=0x7f12a8019690 comp rx=0 tx=0).stop 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 shutdown_connections 2024-09-10T20:26:26.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 --2- 172.21.15.153:0/2855178738 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12b81388e0 0x7f12b813dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 --2- 172.21.15.153:0/2855178738 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12b80a5e20 0x7f12b813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 --2- 172.21.15.153:0/2855178738 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12a403cef0 0x7f12a403f3b0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 --2- 172.21.15.153:0/2855178738 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12b80a5a40 0x7f12b81376f0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 >> 172.21.15.153:0/2855178738 conn(0x7f12b80ab980 msgr2=0x7f12b809ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:26.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 shutdown_connections 2024-09-10T20:26:26.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 -- 172.21.15.153:0/2855178738 wait complete. 2024-09-10T20:26:26.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.185+0000 7f12bea02640 1 librados: shutdown 2024-09-10T20:26:26.209 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-10T20:26:26.209 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:26.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.317+0000 7f7dfad38640 10 monclient: get_monmap_and_config 2024-09-10T20:26:26.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.317+0000 7f7dfad38640 10 monclient: build_initial_monmap 2024-09-10T20:26:26.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.317+0000 7f7dfad38640 10 monclient: monmap: 2024-09-10T20:26:26.325 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:26.325 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.325 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:26.324538+0000 2024-09-10T20:26:26.325 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:26.324538+0000 2024-09-10T20:26:26.325 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:26.325 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.326 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:26.326 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:26.326 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:26.326 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:26.326 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:26.326 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.317+0000 7f7dfad38640 1 Processor -- start 2024-09-10T20:26:26.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.317+0000 7f7dfad38640 1 -- start start 2024-09-10T20:26:26.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.317+0000 7f7dfad38640 10 monclient: init 2024-09-10T20:26:26.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 10 monclient: _add_conns ranks=[3,0,4,1,2] 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7df409ce10 0x7f7df409d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 10 monclient(hunting): picked mon.noname-a con 0x7f7df409ce10 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7df40a0270 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7df409d730 0x7f7df4135c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 10 monclient(hunting): picked mon.noname-c con 0x7f7df409d730 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7df40a3040 con 0x7f7df40a0270 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f7df409ea30 con 0x7f7df409ce10 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7df409f5b0 con 0x7f7df409d730 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:26.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7dfad38640 10 monclient(hunting): authenticate will time out at 946.464204s 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7df409ce10 0x7f7df409d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df3fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f7df40a0270 0x7f7df4135580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60546/0 (socket says 172.21.15.153:60546) 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df3fff640 1 -- 172.21.15.153:0/3810846154 learned_addr learned my addr 172.21.15.153:0/3810846154 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient(hunting): get_auth_request con 0x7f7df409ce10 auth_method 0 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df92d5640 1 --2- 172.21.15.153:0/3810846154 >> v2:172.21.15.176:3300/0 conn(0x7f7df409d730 0x7f7df4135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 1 -- 172.21.15.153:0/3810846154 >> v2:172.21.15.176:3300/0 conn(0x7f7df409d730 msgr2=0x7f7df4135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:26.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 1 --2- 172.21.15.153:0/3810846154 >> v2:172.21.15.176:3300/0 conn(0x7f7df409d730 0x7f7df4135c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 1 -- 172.21.15.153:0/3810846154 >> v1:172.21.15.153:6789/0 conn(0x7f7df40a0270 legacy=0x7f7df4135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:26.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient: _finish_hunting 0 2024-09-10T20:26:26.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 1 monclient: found mon.noname-a 2024-09-10T20:26:26.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:26.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.321+0000 7f7df8ad4640 1 -- 172.21.15.153:0/3810846154 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df409f890 con 0x7f7df409ce10 2024-09-10T20:26:26.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df8ad4640 1 -- 172.21.15.153:0/3810846154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df409ce10 msgr2=0x7f7df409d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:26.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df8ad4640 1 --2- 172.21.15.153:0/3810846154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df409ce10 0x7f7df409d1f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7df406f2f0 tx=0x7f7ddc01c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eac046193f76d128 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 1 -- 172.21.15.153:0/3810846154 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc0ac070 con 0x7f7df409ce10 2024-09-10T20:26:26.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:26.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 10 monclient: dump: 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:26.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 1 -- 172.21.15.153:0/3810846154 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ddc0a6760 con 0x7f7df409ce10 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 1 -- 172.21.15.153:0/3810846154 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ddc0a6b30 con 0x7f7df409ce10 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 5 monclient: authenticate success, global_id 4299 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 4 set_mon_vals no callback set 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df37fe640 10 monclient: dump: 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.336 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient: shutdown 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 -- 172.21.15.153:0/3810846154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df409ce10 msgr2=0x7f7df409d1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 --2- 172.21.15.153:0/3810846154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df409ce10 0x7f7df409d1f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7df406f2f0 tx=0x7f7ddc01c7f0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 -- 172.21.15.153:0/3810846154 shutdown_connections 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 --2- 172.21.15.153:0/3810846154 >> v2:172.21.15.176:3300/0 conn(0x7f7df409d730 0x7f7df4135c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 --2- 172.21.15.153:0/3810846154 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df409ce10 0x7f7df409d1f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 -- 172.21.15.153:0/3810846154 >> 172.21.15.153:0/3810846154 conn(0x7f7df40a71b0 msgr2=0x7f7df40a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 -- 172.21.15.153:0/3810846154 shutdown_connections 2024-09-10T20:26:26.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 -- 172.21.15.153:0/3810846154 wait complete. 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient: build_initial_monmap 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient: monmap: 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 librados: starting msgr at 2024-09-10T20:26:26.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 librados: starting objecter 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 Processor -- start 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 -- start start 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 librados: setting wanted keys 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 librados: calling monclient init 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient: init 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7df409ce10 0x7f7df4140cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient(hunting): picked mon.noname-b con 0x7f7df409ce10 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7df409d730 0x7f7df4141230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient(hunting): picked mon.noname-c con 0x7f7df409d730 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df413a580 0x7f7df41d0380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient(hunting): picked mon.noname-a con 0x7f7df413a580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:26.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7df4136350 con 0x7f7df413a580 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7df41361d0 con 0x7f7df409ce10 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7df4140070 con 0x7f7df409d730 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7dfad38640 10 monclient(hunting): authenticate will time out at 946.470860s 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df413a580 0x7f7df41d0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df413a580 0x7f7df41d0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42400/0 (socket says 172.21.15.153:42400) 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 1 -- 172.21.15.153:0/4212523998 learned_addr learned my addr 172.21.15.153:0/4212523998 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient(hunting): get_auth_request con 0x7f7df413a580 auth_method 0 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df3fff640 1 --2- 172.21.15.153:0/4212523998 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7df409d730 0x7f7df4141230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df3fff640 10 monclient(hunting): get_auth_request con 0x7f7df409d730 auth_method 0 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df8ad4640 1 --2- 172.21.15.153:0/4212523998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7df409ce10 0x7f7df4140cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 1 -- 172.21.15.153:0/4212523998 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7df409d730 msgr2=0x7f7df4141230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 1 --2- 172.21.15.153:0/4212523998 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7df409d730 0x7f7df4141230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 1 -- 172.21.15.153:0/4212523998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7df409ce10 msgr2=0x7f7df4140cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 1 --2- 172.21.15.153:0/4212523998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7df409ce10 0x7f7df4140cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient: _finish_hunting 0 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 1 monclient: found mon.noname-a 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.325+0000 7f7df92d5640 1 -- 172.21.15.153:0/4212523998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7df413a9c0 con 0x7f7df413a580 2024-09-10T20:26:26.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df92d5640 1 -- 172.21.15.153:0/4212523998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df413a580 msgr2=0x7f7df41d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:26.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df92d5640 1 --2- 172.21.15.153:0/4212523998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df413a580 0x7f7df41d0380 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7de400c7f0 tx=0x7f7de400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 client.?.objecter ms_handle_connect 0x7f7df413a580 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 1 -- 172.21.15.153:0/4212523998 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de4015070 con 0x7f7df413a580 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 monclient: dump: 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.345 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 monclient: _finish_auth 0 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 monclient: _check_auth_tickets 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 1 -- 172.21.15.153:0/4212523998 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7de4004220 con 0x7f7df413a580 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 1 -- 172.21.15.153:0/4212523998 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7de4007830 con 0x7f7df413a580 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 monclient: dump: 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.346 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 5 monclient: authenticate success, global_id 4302 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 10 monclient: _renew_subs 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7df4141ce0 con 0x7f7df413a580 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df2ffd640 4 set_mon_vals no callback set 2024-09-10T20:26:26.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 10 monclient: _renew_subs 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7df413b4e0 con 0x7f7df413a580 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 1 librados: init done 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7df409d1f0 con 0x7f7df413a580 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 1 -- 172.21.15.153:0/4212523998 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f7de4007a50 con 0x7f7df413a580 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 1 --2- 172.21.15.153:0/4212523998 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7dd403cef0 0x7f7dd403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 1 -- 172.21.15.153:0/4212523998 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f7de404f2b0 con 0x7f7df413a580 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 10 client.4302.objecter ms_dispatch 0x7f7df4000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:26.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 3 client.4302.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 3 client.4302.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df17fa640 20 client.4302.objecter dump_active .. 0 homeless 2024-09-10T20:26:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df8ad4640 1 --2- 172.21.15.153:0/4212523998 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7dd403cef0 0x7f7dd403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.329+0000 7f7df8ad4640 10 monclient: get_auth_request con 0x7f7dd403cef0 auth_method 0 2024-09-10T20:26:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.333+0000 7f7df17fa640 10 client.4302.objecter ms_handle_connect 0x7f7dd403cef0 2024-09-10T20:26:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.333+0000 7f7df8ad4640 1 --2- 172.21.15.153:0/4212523998 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7dd403cef0 0x7f7dd403f3b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7df409f200 tx=0x7f7ddc0a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.333+0000 7f7df17fa640 1 -- 172.21.15.153:0/4212523998 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f7de4007d40 con 0x7f7df413a580 2024-09-10T20:26:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.333+0000 7f7df17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.333+0000 7f7df17fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:26.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.489+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7df413b130 con 0x7f7dd403cef0 2024-09-10T20:26:26.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7df17fa640 1 -- 172.21.15.153:0/4212523998 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34165 (secure 0 0 0) 0x7f7df413b130 con 0x7f7dd403cef0 2024-09-10T20:26:26.499 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:26.500 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-10T20:26:26.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 10 librados: watch_flush enter 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 10 librados: watch_flush exit 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 20 client.4302.objecter shutdown clearing up homeless session... 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 10 client.4302.objecter successfully canceled tick 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7dd403cef0 msgr2=0x7f7dd403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 --2- 172.21.15.153:0/4212523998 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7dd403cef0 0x7f7dd403f3b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f7df409f200 tx=0x7f7ddc0a7000 comp rx=0 tx=0).stop 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 10 monclient: shutdown 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df413a580 msgr2=0x7f7df41d0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 --2- 172.21.15.153:0/4212523998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df413a580 0x7f7df41d0380 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7de400c7f0 tx=0x7f7de400ccb0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 shutdown_connections 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 --2- 172.21.15.153:0/4212523998 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7df413a580 0x7f7df41d0380 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 --2- 172.21.15.153:0/4212523998 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7dd403cef0 0x7f7dd403f3b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 --2- 172.21.15.153:0/4212523998 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7df409d730 0x7f7df4141230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 --2- 172.21.15.153:0/4212523998 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7df409ce10 0x7f7df4140cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 >> 172.21.15.153:0/4212523998 conn(0x7f7df40a71b0 msgr2=0x7f7df4074220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:26.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 shutdown_connections 2024-09-10T20:26:26.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 -- 172.21.15.153:0/4212523998 wait complete. 2024-09-10T20:26:26.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.493+0000 7f7dfad38640 1 librados: shutdown 2024-09-10T20:26:26.516 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-10T20:26:25.070079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163316,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067724,"statfs":{"total":579820584960,"available":579653349376,"internally_reserved":0,"allocated":1163264,"data_stored":1383940,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001817"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773251+0000","last_change":"2024-09-10T20:26:21.773251+0000","last_active":"2024-09-10T20:26:21.773251+0000","last_peered":"2024-09-10T20:26:21.773251+0000","last_clean":"2024-09-10T20:26:21.773251+0000","last_became_active":"2024-09-10T20:26:19.356844+0000","last_became_peered":"2024-09-10T20:26:19.356844+0000","last_unstale":"2024-09-10T20:26:21.773251+0000","last_undegraded":"2024-09-10T20:26:21.773251+0000","last_fullsized":"2024-09-10T20:26:21.773251+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:23.413224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.381255+0000","last_change":"2024-09-10T20:26:21.381255+0000","last_active":"2024-09-10T20:26:21.381255+0000","last_peered":"2024-09-10T20:26:21.381255+0000","last_clean":"2024-09-10T20:26:21.381255+0000","last_became_active":"2024-09-10T20:26:19.363220+0000","last_became_peered":"2024-09-10T20:26:19.363220+0000","last_unstale":"2024-09-10T20:26:21.381255+0000","last_undegraded":"2024-09-10T20:26:21.381255+0000","last_fullsized":"2024-09-10T20:26:21.381255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:09.279291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096679600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773020+0000","last_change":"2024-09-10T20:26:21.773020+0000","last_active":"2024-09-10T20:26:21.773020+0000","last_peered":"2024-09-10T20:26:21.773020+0000","last_clean":"2024-09-10T20:26:21.773020+0000","last_became_active":"2024-09-10T20:26:19.363294+0000","last_became_peered":"2024-09-10T20:26:19.363294+0000","last_unstale":"2024-09-10T20:26:21.773020+0000","last_undegraded":"2024-09-10T20:26:21.773020+0000","last_fullsized":"2024-09-10T20:26:21.773020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:54.300619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053952999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.378377+0000","last_change":"2024-09-10T20:26:21.378377+0000","last_active":"2024-09-10T20:26:21.378377+0000","last_peered":"2024-09-10T20:26:21.378377+0000","last_clean":"2024-09-10T20:26:21.378377+0000","last_became_active":"2024-09-10T20:26:19.359961+0000","last_became_peered":"2024-09-10T20:26:19.359961+0000","last_unstale":"2024-09-10T20:26:21.378377+0000","last_undegraded":"2024-09-10T20:26:21.378377+0000","last_fullsized":"2024-09-10T20:26:21.378377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:21:01.940376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.394496+0000","last_change":"2024-09-10T20:26:21.386697+0000","last_active":"2024-09-10T20:26:21.394496+0000","last_peered":"2024-09-10T20:26:21.394496+0000","last_clean":"2024-09-10T20:26:21.394496+0000","last_became_active":"2024-09-10T20:26:19.360427+0000","last_became_peered":"2024-09-10T20:26:19.360427+0000","last_unstale":"2024-09-10T20:26:21.394496+0000","last_undegraded":"2024-09-10T20:26:21.394496+0000","last_fullsized":"2024-09-10T20:26:21.394496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:53.136006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020581079999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.575187+0000","last_change":"2024-09-10T20:26:21.575187+0000","last_active":"2024-09-10T20:26:21.575187+0000","last_peered":"2024-09-10T20:26:21.575187+0000","last_clean":"2024-09-10T20:26:21.575187+0000","last_became_active":"2024-09-10T20:26:19.362858+0000","last_became_peered":"2024-09-10T20:26:19.362858+0000","last_unstale":"2024-09-10T20:26:21.575187+0000","last_undegraded":"2024-09-10T20:26:21.575187+0000","last_fullsized":"2024-09-10T20:26:21.575187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:22:17.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078714099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.378915+0000","last_change":"2024-09-10T20:26:21.378915+0000","last_active":"2024-09-10T20:26:21.378915+0000","last_peered":"2024-09-10T20:26:21.378915+0000","last_clean":"2024-09-10T20:26:21.378915+0000","last_became_active":"2024-09-10T20:26:19.362158+0000","last_became_peered":"2024-09-10T20:26:19.362158+0000","last_unstale":"2024-09-10T20:26:21.378915+0000","last_undegraded":"2024-09-10T20:26:21.378915+0000","last_fullsized":"2024-09-10T20:26:21.378915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:50.708554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010122729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773025+0000","last_change":"2024-09-10T20:26:21.773025+0000","last_active":"2024-09-10T20:26:21.773025+0000","last_peered":"2024-09-10T20:26:21.773025+0000","last_clean":"2024-09-10T20:26:21.773025+0000","last_became_active":"2024-09-10T20:26:19.359775+0000","last_became_peered":"2024-09-10T20:26:19.359775+0000","last_unstale":"2024-09-10T20:26:21.773025+0000","last_undegraded":"2024-09-10T20:26:21.773025+0000","last_fullsized":"2024-09-10T20:26:21.773025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:35.427907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054594699999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.772391+0000","last_change":"2024-09-10T20:26:14.579551+0000","last_active":"2024-09-10T20:26:21.772391+0000","last_peered":"2024-09-10T20:26:21.772391+0000","last_clean":"2024-09-10T20:26:21.772391+0000","last_became_active":"2024-09-10T20:26:14.577602+0000","last_became_peered":"2024-09-10T20:26:14.577602+0000","last_unstale":"2024-09-10T20:26:21.772391+0000","last_undegraded":"2024-09-10T20:26:21.772391+0000","last_fullsized":"2024-09-10T20:26:21.772391+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_clean_scrub_stamp":"2024-09-10T20:26:13.287430+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:41.463707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":583269,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":583269,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":54341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:26.517 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.613+0000 7fdd16b0e640 10 monclient: get_monmap_and_config 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.613+0000 7fdd16b0e640 10 monclient: build_initial_monmap 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.613+0000 7fdd16b0e640 10 monclient: monmap: 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:26.621388+0000 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:26.621388+0000 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:26.622 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:26.623 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:26.623 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:26.623 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 1 Processor -- start 2024-09-10T20:26:26.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 1 -- start start 2024-09-10T20:26:26.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 10 monclient: init 2024-09-10T20:26:26.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:26.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 10 monclient: _add_conns ranks=[0,1,4,2,3] 2024-09-10T20:26:26.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd101364f0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:26.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdd101360a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:26.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fdd100a1990 0x7fdd1013eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fdd100a1990 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd100a27a0 con 0x7fdd101364f0 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd100a2a80 con 0x7fdd101360a0 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fdd100a2550 con 0x7fdd100a1990 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd16b0e640 10 monclient(hunting): authenticate will time out at 946.761015s 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd148aa640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdd101364f0 0x7fdd101368d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60552/0 (socket says 172.21.15.153:60552) 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd148aa640 1 -- 172.21.15.153:0/714303950 learned_addr learned my addr 172.21.15.153:0/714303950 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 1 --2- 172.21.15.153:0/714303950 >> v2:172.21.15.176:3300/0 conn(0x7fdd100a1990 0x7fdd1013eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient(hunting): get_auth_request con 0x7fdd100a1990 auth_method 0 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient(hunting): handle_auth_done global_id 4270 payload 274 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 1 -- 172.21.15.153:0/714303950 >> v1:172.21.15.176:6789/0 conn(0x7fdd101360a0 legacy=0x7fdd1013b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 1 -- 172.21.15.153:0/714303950 >> v1:172.21.15.153:6789/0 conn(0x7fdd101364f0 legacy=0x7fdd101368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient: _finish_hunting 0 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 1 monclient: found mon.noname-c 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:26:26.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.617+0000 7fdd150ab640 1 -- 172.21.15.153:0/714303950 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd100a2d60 con 0x7fdd100a1990 2024-09-10T20:26:26.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd150ab640 1 -- 172.21.15.153:0/714303950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 msgr2=0x7fdd1013eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:26.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd150ab640 1 --2- 172.21.15.153:0/714303950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 0x7fdd1013eb50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009d90 tx=0x7fdd0801c830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=669ae276eaaa0ce4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 1 -- 172.21.15.153:0/714303950 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd080b2070 con 0x7fdd100a1990 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 10 monclient: dump: 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.629 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 1 -- 172.21.15.153:0/714303950 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd080a5970 con 0x7fdd100a1990 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 1 -- 172.21.15.153:0/714303950 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd080a5d40 con 0x7fdd100a1990 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd0f7fe640 10 monclient: dump: 2024-09-10T20:26:26.630 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 5 monclient: authenticate success, global_id 4270 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:26.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 4 set_mon_vals no callback set 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 10 monclient: shutdown 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 1 -- 172.21.15.153:0/714303950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 msgr2=0x7fdd1013eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 1 --2- 172.21.15.153:0/714303950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 0x7fdd1013eb50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009d90 tx=0x7fdd0801c830 comp rx=0 tx=0).stop 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 1 -- 172.21.15.153:0/714303950 shutdown_connections 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 1 --2- 172.21.15.153:0/714303950 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 0x7fdd1013eb50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 1 -- 172.21.15.153:0/714303950 >> 172.21.15.153:0/714303950 conn(0x7fdd100a71b0 msgr2=0x7fdd100a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 1 -- 172.21.15.153:0/714303950 shutdown_connections 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.621+0000 7fdd16b0e640 1 -- 172.21.15.153:0/714303950 wait complete. 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient: build_initial_monmap 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient: monmap: 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:26.637 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 librados: starting msgr at 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 librados: starting objecter 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 Processor -- start 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 -- start start 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 librados: setting wanted keys 2024-09-10T20:26:26.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 librados: calling monclient init 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient: init 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 0x7fdd1013e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient(hunting): picked mon.noname-b con 0x7fdd100a1990 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd101360a0 0x7fdd1013e650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient(hunting): picked mon.noname-c con 0x7fdd101360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd101364f0 0x7fdd101370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient(hunting): picked mon.noname-a con 0x7fdd101364f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd10141d80 con 0x7fdd101364f0 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd10141c00 con 0x7fdd100a1990 2024-09-10T20:26:26.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdd10141f00 con 0x7fdd101360a0 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd16b0e640 10 monclient(hunting): authenticate will time out at 946.769285s 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd101364f0 0x7fdd101370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd101364f0 0x7fdd101370a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42416/0 (socket says 172.21.15.153:42416) 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd0ffff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd101360a0 0x7fdd1013e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 1 -- 172.21.15.153:0/1040427930 learned_addr learned my addr 172.21.15.153:0/1040427930 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient(hunting): get_auth_request con 0x7fdd101364f0 auth_method 0 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd0ffff640 10 monclient(hunting): get_auth_request con 0x7fdd101360a0 auth_method 0 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd0ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd0ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd0ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd0ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd0ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd0ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient(hunting): handle_auth_done global_id 4305 payload 824 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd148aa640 1 --2- 172.21.15.153:0/1040427930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 0x7fdd1013e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 1 -- 172.21.15.153:0/1040427930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd101360a0 msgr2=0x7fdd1013e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 1 --2- 172.21.15.153:0/1040427930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd101360a0 0x7fdd1013e650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 1 -- 172.21.15.153:0/1040427930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 msgr2=0x7fdd1013e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 1 --2- 172.21.15.153:0/1040427930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 0x7fdd1013e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient: _finish_hunting 0 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 1 monclient: found mon.noname-a 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd150ab640 1 -- 172.21.15.153:0/1040427930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd1009e0a0 con 0x7fdd101364f0 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.625+0000 7fdd0ffff640 1 --2- 172.21.15.153:0/1040427930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd101360a0 0x7fdd1013e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd150ab640 1 -- 172.21.15.153:0/1040427930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd101364f0 msgr2=0x7fdd101370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd150ab640 1 --2- 172.21.15.153:0/1040427930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd101364f0 0x7fdd101370a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009d90 tx=0x7fdd080195d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 client.?.objecter ms_handle_connect 0x7fdd101364f0 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 1 -- 172.21.15.153:0/1040427930 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd080bd020 con 0x7fdd101364f0 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:26.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 monclient: dump: 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 monclient: _finish_auth 0 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 monclient: _check_auth_tickets 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 1 -- 172.21.15.153:0/1040427930 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd080adb00 con 0x7fdd101364f0 2024-09-10T20:26:26.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 1 -- 172.21.15.153:0/1040427930 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd080b2040 con 0x7fdd101364f0 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 5 monclient: authenticate success, global_id 4305 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 10 monclient: _renew_subs 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd1013a600 con 0x7fdd101364f0 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0effd640 4 set_mon_vals no callback set 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 monclient: dump: 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.644 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 10 client.4305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 10 monclient: _renew_subs 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd1013eb90 con 0x7fdd101364f0 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 1 librados: init done 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd101368d0 con 0x7fdd101364f0 2024-09-10T20:26:26.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 1 -- 172.21.15.153:0/1040427930 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fdd080c0020 con 0x7fdd101364f0 2024-09-10T20:26:26.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 1 --2- 172.21.15.153:0/1040427930 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdcf403cea0 0x7fdcf403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 1 -- 172.21.15.153:0/1040427930 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7fdd080ea4d0 con 0x7fdd101364f0 2024-09-10T20:26:26.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 10 client.4305.objecter ms_dispatch 0x7fdd10000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:26.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 3 client.4305.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:26.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 3 client.4305.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:26.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd0d7fa640 20 client.4305.objecter dump_active .. 0 homeless 2024-09-10T20:26:26.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd148aa640 1 --2- 172.21.15.153:0/1040427930 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdcf403cea0 0x7fdcf403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.629+0000 7fdd148aa640 10 monclient: get_auth_request con 0x7fdcf403cea0 auth_method 0 2024-09-10T20:26:26.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.633+0000 7fdd148aa640 1 --2- 172.21.15.153:0/1040427930 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdcf403cea0 0x7fdcf403f360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdd04005060 tx=0x7fdd04009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.633+0000 7fdd0d7fa640 10 client.4305.objecter ms_handle_connect 0x7fdcf403cea0 2024-09-10T20:26:26.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.633+0000 7fdd0d7fa640 1 -- 172.21.15.153:0/1040427930 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fdd080c7070 con 0x7fdd101364f0 2024-09-10T20:26:26.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.633+0000 7fdd0d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.633+0000 7fdd0d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:26.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.785+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdd10131240 con 0x7fdcf403cea0 2024-09-10T20:26:26.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.785+0000 7fdd0d7fa640 1 -- 172.21.15.153:0/1040427930 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34165 (secure 0 0 0) 0x7fdd040070d0 con 0x7fdcf403cea0 2024-09-10T20:26:26.795 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:26:26.797 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-10T20:26:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 10 librados: watch_flush enter 2024-09-10T20:26:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 10 librados: watch_flush exit 2024-09-10T20:26:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 20 client.4305.objecter shutdown clearing up homeless session... 2024-09-10T20:26:26.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 10 client.4305.objecter successfully canceled tick 2024-09-10T20:26:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdcf403cea0 msgr2=0x7fdcf403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 --2- 172.21.15.153:0/1040427930 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdcf403cea0 0x7fdcf403f360 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdd04005060 tx=0x7fdd04009040 comp rx=0 tx=0).stop 2024-09-10T20:26:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 10 monclient: shutdown 2024-09-10T20:26:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:26.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd101364f0 msgr2=0x7fdd101370a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 --2- 172.21.15.153:0/1040427930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd101364f0 0x7fdd101370a0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fdd08009d90 tx=0x7fdd080195d0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 shutdown_connections 2024-09-10T20:26:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 --2- 172.21.15.153:0/1040427930 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd101364f0 0x7fdd101370a0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 --2- 172.21.15.153:0/1040427930 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdcf403cea0 0x7fdcf403f360 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 --2- 172.21.15.153:0/1040427930 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd101360a0 0x7fdd1013e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.800 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 --2- 172.21.15.153:0/1040427930 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd100a1990 0x7fdd1013e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 >> 172.21.15.153:0/1040427930 conn(0x7fdd100a71b0 msgr2=0x7fdd100a56b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:26.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 shutdown_connections 2024-09-10T20:26:26.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 -- 172.21.15.153:0/1040427930 wait complete. 2024-09-10T20:26:26.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.789+0000 7fdd16b0e640 1 librados: shutdown 2024-09-10T20:26:26.811 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-09-10T20:26:25.070079+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163316,"kb_used_data":1136,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067724,"statfs":{"total":579820584960,"available":579653349376,"internally_reserved":0,"allocated":1163264,"data_stored":1383940,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9567,"internal_metadata":165927585},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001817"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773251+0000","last_change":"2024-09-10T20:26:21.773251+0000","last_active":"2024-09-10T20:26:21.773251+0000","last_peered":"2024-09-10T20:26:21.773251+0000","last_clean":"2024-09-10T20:26:21.773251+0000","last_became_active":"2024-09-10T20:26:19.356844+0000","last_became_peered":"2024-09-10T20:26:19.356844+0000","last_unstale":"2024-09-10T20:26:21.773251+0000","last_undegraded":"2024-09-10T20:26:21.773251+0000","last_fullsized":"2024-09-10T20:26:21.773251+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:23.413224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.381255+0000","last_change":"2024-09-10T20:26:21.381255+0000","last_active":"2024-09-10T20:26:21.381255+0000","last_peered":"2024-09-10T20:26:21.381255+0000","last_clean":"2024-09-10T20:26:21.381255+0000","last_became_active":"2024-09-10T20:26:19.363220+0000","last_became_peered":"2024-09-10T20:26:19.363220+0000","last_unstale":"2024-09-10T20:26:21.381255+0000","last_undegraded":"2024-09-10T20:26:21.381255+0000","last_fullsized":"2024-09-10T20:26:21.381255+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:09.279291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096679600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773020+0000","last_change":"2024-09-10T20:26:21.773020+0000","last_active":"2024-09-10T20:26:21.773020+0000","last_peered":"2024-09-10T20:26:21.773020+0000","last_clean":"2024-09-10T20:26:21.773020+0000","last_became_active":"2024-09-10T20:26:19.363294+0000","last_became_peered":"2024-09-10T20:26:19.363294+0000","last_unstale":"2024-09-10T20:26:21.773020+0000","last_undegraded":"2024-09-10T20:26:21.773020+0000","last_fullsized":"2024-09-10T20:26:21.773020+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:54.300619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053952999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.378377+0000","last_change":"2024-09-10T20:26:21.378377+0000","last_active":"2024-09-10T20:26:21.378377+0000","last_peered":"2024-09-10T20:26:21.378377+0000","last_clean":"2024-09-10T20:26:21.378377+0000","last_became_active":"2024-09-10T20:26:19.359961+0000","last_became_peered":"2024-09-10T20:26:19.359961+0000","last_unstale":"2024-09-10T20:26:21.378377+0000","last_undegraded":"2024-09-10T20:26:21.378377+0000","last_fullsized":"2024-09-10T20:26:21.378377+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:21:01.940376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.394496+0000","last_change":"2024-09-10T20:26:21.386697+0000","last_active":"2024-09-10T20:26:21.394496+0000","last_peered":"2024-09-10T20:26:21.394496+0000","last_clean":"2024-09-10T20:26:21.394496+0000","last_became_active":"2024-09-10T20:26:19.360427+0000","last_became_peered":"2024-09-10T20:26:19.360427+0000","last_unstale":"2024-09-10T20:26:21.394496+0000","last_undegraded":"2024-09-10T20:26:21.394496+0000","last_fullsized":"2024-09-10T20:26:21.394496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:53.136006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020581079999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.575187+0000","last_change":"2024-09-10T20:26:21.575187+0000","last_active":"2024-09-10T20:26:21.575187+0000","last_peered":"2024-09-10T20:26:21.575187+0000","last_clean":"2024-09-10T20:26:21.575187+0000","last_became_active":"2024-09-10T20:26:19.362858+0000","last_became_peered":"2024-09-10T20:26:19.362858+0000","last_unstale":"2024-09-10T20:26:21.575187+0000","last_undegraded":"2024-09-10T20:26:21.575187+0000","last_fullsized":"2024-09-10T20:26:21.575187+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:22:17.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078714099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.378915+0000","last_change":"2024-09-10T20:26:21.378915+0000","last_active":"2024-09-10T20:26:21.378915+0000","last_peered":"2024-09-10T20:26:21.378915+0000","last_clean":"2024-09-10T20:26:21.378915+0000","last_became_active":"2024-09-10T20:26:19.362158+0000","last_became_peered":"2024-09-10T20:26:19.362158+0000","last_unstale":"2024-09-10T20:26:21.378915+0000","last_undegraded":"2024-09-10T20:26:21.378915+0000","last_fullsized":"2024-09-10T20:26:21.378915+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:50.708554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010122729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":21,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.773025+0000","last_change":"2024-09-10T20:26:21.773025+0000","last_active":"2024-09-10T20:26:21.773025+0000","last_peered":"2024-09-10T20:26:21.773025+0000","last_clean":"2024-09-10T20:26:21.773025+0000","last_became_active":"2024-09-10T20:26:19.359775+0000","last_became_peered":"2024-09-10T20:26:19.359775+0000","last_unstale":"2024-09-10T20:26:21.773025+0000","last_undegraded":"2024-09-10T20:26:21.773025+0000","last_fullsized":"2024-09-10T20:26:21.773025+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:35.427907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054594699999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'72","reported_seq":111,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-10T20:26:21.772391+0000","last_change":"2024-09-10T20:26:14.579551+0000","last_active":"2024-09-10T20:26:21.772391+0000","last_peered":"2024-09-10T20:26:21.772391+0000","last_clean":"2024-09-10T20:26:21.772391+0000","last_became_active":"2024-09-10T20:26:14.577602+0000","last_became_peered":"2024-09-10T20:26:14.577602+0000","last_unstale":"2024-09-10T20:26:21.772391+0000","last_undegraded":"2024-09-10T20:26:21.772391+0000","last_fullsized":"2024-09-10T20:26:21.772391+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_clean_scrub_stamp":"2024-09-10T20:26:13.287430+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:41.463707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":172032,"data_stored":54341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27264,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344576,"statfs":{"total":96636764160,"available":96608845824,"internally_reserved":0,"allocated":233472,"data_stored":583269,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640261,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27256,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344584,"statfs":{"total":96636764160,"available":96608854016,"internally_reserved":0,"allocated":233472,"data_stored":583269,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607557,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":172,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":176128,"data_stored":54360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":168,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":172032,"data_stored":54341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:26:26.812 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-10T20:26:26.813 DEBUG:tasks.mon_thrash:Yielding 2024-09-10T20:26:26.813 INFO:teuthology.run_tasks:Running task workunit... 2024-09-10T20:26:26.825 INFO:tasks.workunit:Pulling workunits from ref bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:26:26.825 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-10T20:26:26.826 DEBUG:teuthology.orchestra.run.smithi176:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-10T20:26:26.826 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-10T20:26:26.826 INFO:tasks.mon_thrash.mon_thrasher:seed: 1725999986, revive delay: 90.0, thrash delay: 1.0 thrash many: True, maintain quorum: True store thrash: True, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-10T20:26:26.826 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T20:26:26.826 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:26:26.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:26:26.833 INFO:teuthology.orchestra.run.smithi176.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-10T20:26:26.833 DEBUG:teuthology.orchestra.run.smithi176:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T20:26:26.881 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-10T20:26:26.882 DEBUG:teuthology.orchestra.run.smithi176:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-10T20:26:26.929 INFO:tasks.workunit:timeout=3h 2024-09-10T20:26:26.929 INFO:tasks.workunit:cleanup=True 2024-09-10T20:26:26.930 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout bad1f558c327094897e4a32e99dd5cb4ae6ccd46 2024-09-10T20:26:26.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient: get_monmap_and_config 2024-09-10T20:26:26.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient: build_initial_monmap 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient: monmap: 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:26.932239+0000 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:26.932239+0000 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:26.933 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 1 Processor -- start 2024-09-10T20:26:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 1 -- start start 2024-09-10T20:26:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient: init 2024-09-10T20:26:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient: _add_conns ranks=[3,1,0,4,2] 2024-09-10T20:26:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2eac0a41a0 0x7f2eac0a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient(hunting): picked mon.noname-a con 0x7f2eac0a41a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:26.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2eac0a4580 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2eac0a49d0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eac0a67f0 con 0x7f2eac0a49d0 2024-09-10T20:26:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2eac0a6a20 con 0x7f2eac0a4580 2024-09-10T20:26:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f2eac0a2570 con 0x7f2eac0a41a0 2024-09-10T20:26:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.925+0000 7f2eb2082640 10 monclient(hunting): authenticate will time out at 947.071858s 2024-09-10T20:26:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f2eac0a41a0 0x7f2eac0a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eabfff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2eac0a49d0 0x7f2eac0a60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60556/0 (socket says 172.21.15.153:60556) 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eabfff640 1 -- 172.21.15.153:0/239491177 learned_addr learned my addr 172.21.15.153:0/239491177 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient(hunting): get_auth_request con 0x7f2eac0a41a0 auth_method 0 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient(hunting): handle_auth_done global_id 4308 payload 274 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 1 -- 172.21.15.153:0/239491177 >> v1:172.21.15.176:6789/0 conn(0x7f2eac0a4580 legacy=0x7f2eac0a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:26.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 1 -- 172.21.15.153:0/239491177 >> v1:172.21.15.153:6789/0 conn(0x7f2eac0a49d0 legacy=0x7f2eac0a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 1 monclient: found mon.noname-a 2024-09-10T20:26:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.929+0000 7f2eab7fe640 1 -- 172.21.15.153:0/239491177 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac0a3c80 con 0x7f2eac0a41a0 2024-09-10T20:26:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eab7fe640 1 -- 172.21.15.153:0/239491177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a41a0 msgr2=0x7f2eac0a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:26.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eab7fe640 1 --2- 172.21.15.153:0/239491177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a41a0 0x7f2eac0a3740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00097c0 tx=0x7f2ea001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eeb263bd489e9442 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 1 -- 172.21.15.153:0/239491177 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00b0070 con 0x7f2eac0a41a0 2024-09-10T20:26:26.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:26.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 10 monclient: dump: 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 10 monclient: _finish_auth 0 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 1 -- 172.21.15.153:0/239491177 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea00ab600 con 0x7f2eac0a41a0 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 1 -- 172.21.15.153:0/239491177 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00ab9d0 con 0x7f2eac0a41a0 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eaa7fc640 10 monclient: dump: 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.943 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.944 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.944 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.944 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.944 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 5 monclient: authenticate success, global_id 4308 2024-09-10T20:26:26.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:26.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 4 set_mon_vals no callback set 2024-09-10T20:26:26.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 10 monclient: shutdown 2024-09-10T20:26:26.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:26.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 1 -- 172.21.15.153:0/239491177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a41a0 msgr2=0x7f2eac0a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 1 --2- 172.21.15.153:0/239491177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a41a0 0x7f2eac0a3740 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00097c0 tx=0x7f2ea001c360 comp rx=0 tx=0).stop 2024-09-10T20:26:26.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 1 -- 172.21.15.153:0/239491177 shutdown_connections 2024-09-10T20:26:26.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 1 --2- 172.21.15.153:0/239491177 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a41a0 0x7f2eac0a3740 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 1 -- 172.21.15.153:0/239491177 >> 172.21.15.153:0/239491177 conn(0x7f2eac0ab7e0 msgr2=0x7f2eac0a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.933+0000 7f2eb2082640 1 -- 172.21.15.153:0/239491177 shutdown_connections 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 -- 172.21.15.153:0/239491177 wait complete. 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient: build_initial_monmap 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient: monmap: 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:26.948 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 librados: starting msgr at 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 librados: starting objecter 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 Processor -- start 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 -- start start 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 librados: setting wanted keys 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 librados: calling monclient init 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient: init 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:26.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2eac0a41a0 0x7f2eac128a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient(hunting): picked mon.noname-c con 0x7f2eac0a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eac0a4580 0x7f2eac140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient(hunting): picked mon.noname-b con 0x7f2eac0a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a49d0 0x7f2eac1d0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient(hunting): picked mon.noname-a con 0x7f2eac0a49d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac144630 con 0x7f2eac0a49d0 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2eac070400 con 0x7f2eac0a4580 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2eac1447b0 con 0x7f2eac0a41a0 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:26.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eb2082640 10 monclient(hunting): authenticate will time out at 947.081441s 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a49d0 0x7f2eac1d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2eac0a41a0 0x7f2eac128a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2eac0a41a0 0x7f2eac128a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55856/0 (socket says 172.21.15.153:55856) 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient(hunting): get_auth_request con 0x7f2eac0a49d0 auth_method 0 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 1 -- 172.21.15.153:0/3014851248 learned_addr learned my addr 172.21.15.153:0/3014851248 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 10 monclient(hunting): get_auth_request con 0x7f2eac0a41a0 auth_method 0 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:26.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient(hunting): handle_auth_done global_id 4311 payload 824 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 1 -- 172.21.15.153:0/3014851248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2eac0a41a0 msgr2=0x7f2eac128a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 1 --2- 172.21.15.153:0/3014851248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2eac0a41a0 0x7f2eac128a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 1 -- 172.21.15.153:0/3014851248 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eac0a4580 msgr2=0x7f2eac140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 1 --2- 172.21.15.153:0/3014851248 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eac0a4580 0x7f2eac140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 1 monclient: found mon.noname-a 2024-09-10T20:26:26.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:26.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.937+0000 7f2eabfff640 1 -- 172.21.15.153:0/3014851248 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eac129f20 con 0x7f2eac0a49d0 2024-09-10T20:26:26.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eabfff640 1 -- 172.21.15.153:0/3014851248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a49d0 msgr2=0x7f2eac1d0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:26.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eabfff640 1 --2- 172.21.15.153:0/3014851248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a49d0 0x7f2eac1d0990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400caf0 tx=0x7f2ea400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 1 -- 172.21.15.153:0/3014851248 DispatchQueue::entry inject delay of 0.002000 2024-09-10T20:26:26.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 client.?.objecter ms_handle_connect 0x7f2eac0a49d0 2024-09-10T20:26:26.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 1 -- 172.21.15.153:0/3014851248 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea400ade0 con 0x7f2eac0a49d0 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 monclient: dump: 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.956 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 monclient: _finish_auth 0 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 5 monclient: authenticate success, global_id 4311 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 1 -- 172.21.15.153:0/3014851248 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea4012400 con 0x7f2eac0a49d0 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 1 -- 172.21.15.153:0/3014851248 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea40127f0 con 0x7f2eac0a49d0 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:26.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 monclient: dump: 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2ea9ffb640 4 set_mon_vals no callback set 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 10 monclient: _renew_subs 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eac0a21c0 con 0x7f2eac0a49d0 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 10 client.4311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 10 monclient: _renew_subs 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eac127820 con 0x7f2eac0a49d0 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 1 librados: init done 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eac0a3740 con 0x7f2eac0a49d0 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 1 -- 172.21.15.153:0/3014851248 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f2ea4012990 con 0x7f2eac0a49d0 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 1 --2- 172.21.15.153:0/3014851248 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2e9803cf20 0x7f2e9803f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 1 -- 172.21.15.153:0/3014851248 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f2ea404f800 con 0x7f2eac0a49d0 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 10 client.4311.objecter ms_dispatch 0x7f2eac000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:26.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 3 client.4311.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.941+0000 7f2e93fff640 3 client.4311.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.945+0000 7f2e93fff640 20 client.4311.objecter dump_active .. 0 homeless 2024-09-10T20:26:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.945+0000 7f2eab7fe640 1 --2- 172.21.15.153:0/3014851248 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2e9803cf20 0x7f2e9803f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.945+0000 7f2eab7fe640 10 monclient: get_auth_request con 0x7f2e9803cf20 auth_method 0 2024-09-10T20:26:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.945+0000 7f2eab7fe640 1 --2- 172.21.15.153:0/3014851248 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2e9803cf20 0x7f2e9803f3e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00097c0 tx=0x7f2ea0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.945+0000 7f2e93fff640 10 client.4311.objecter ms_handle_connect 0x7f2e9803cf20 2024-09-10T20:26:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.949+0000 7f2e93fff640 1 -- 172.21.15.153:0/3014851248 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f2ea401a360 con 0x7f2eac0a49d0 2024-09-10T20:26:26.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.949+0000 7f2e93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:26.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:26.949+0000 7f2e93fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:26.979 INFO:tasks.workunit.client.0.smithi176.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-10T20:26:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.173+0000 7f2eb2082640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:26:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.173+0000 7f2eb2082640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:26:27.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.173+0000 7f2eb2082640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.173+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2eac06cd40 con 0x7f2eac0a49d0 2024-09-10T20:26:27.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.173+0000 7f2e93fff640 1 -- 172.21.15.153:0/3014851248 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f2ea4007a90 con 0x7f2eac0a49d0 2024-09-10T20:26:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.173+0000 7f2e93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:26:27.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.173+0000 7f2e93fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:27.182 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:26:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 10 librados: watch_flush enter 2024-09-10T20:26:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 10 librados: watch_flush exit 2024-09-10T20:26:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 20 client.4311.objecter shutdown clearing up homeless session... 2024-09-10T20:26:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 10 client.4311.objecter successfully canceled tick 2024-09-10T20:26:27.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2e9803cf20 msgr2=0x7f2e9803f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 --2- 172.21.15.153:0/3014851248 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2e9803cf20 0x7f2e9803f3e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2ea00097c0 tx=0x7f2ea0008040 comp rx=0 tx=0).stop 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 10 monclient: shutdown 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a49d0 msgr2=0x7f2eac1d0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 --2- 172.21.15.153:0/3014851248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a49d0 0x7f2eac1d0990 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f2ea400caf0 tx=0x7f2ea400cef0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 shutdown_connections 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 --2- 172.21.15.153:0/3014851248 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2eac0a49d0 0x7f2eac1d0990 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 --2- 172.21.15.153:0/3014851248 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2eac0a4580 0x7f2eac140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 --2- 172.21.15.153:0/3014851248 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2e9803cf20 0x7f2e9803f3e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 --2- 172.21.15.153:0/3014851248 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2eac0a41a0 0x7f2eac128a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 >> 172.21.15.153:0/3014851248 conn(0x7f2eac0ab7e0 msgr2=0x7f2eac0a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 shutdown_connections 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 -- 172.21.15.153:0/3014851248 wait complete. 2024-09-10T20:26:27.184 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.177+0000 7f2eb2082640 1 librados: shutdown 2024-09-10T20:26:27.198 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T20:26:27.198 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T20:26:27.198 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T20:26:27.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:26:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.297+0000 7f0a02401640 10 monclient: get_monmap_and_config 2024-09-10T20:26:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.297+0000 7f0a02401640 10 monclient: build_initial_monmap 2024-09-10T20:26:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.297+0000 7f0a02401640 10 monclient: monmap: 2024-09-10T20:26:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:27.306 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:27.305788+0000 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:27.305788+0000 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:27.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 1 Processor -- start 2024-09-10T20:26:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 1 -- start start 2024-09-10T20:26:27.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 10 monclient: init 2024-09-10T20:26:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 10 monclient: _add_conns ranks=[1,3,0,2,4] 2024-09-10T20:26:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f09fc0a5ab0 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f09fc0a65a0 0x7f09fc13dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 10 monclient(hunting): picked mon.noname-a con 0x7f09fc0a65a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f09fc13e1b0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:27.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09fc070780 con 0x7f09fc13e1b0 2024-09-10T20:26:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09fc0709c0 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f09fc0a3010 con 0x7f09fc0a65a0 2024-09-10T20:26:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a02401640 10 monclient(hunting): authenticate will time out at 947.445378s 2024-09-10T20:26:27.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f09fc0a65a0 0x7f09fc13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a0099e640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f09fc13e1b0 0x7f09fc1405a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60560/0 (socket says 172.21.15.153:60560) 2024-09-10T20:26:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f0a0099e640 1 -- 172.21.15.153:0/2421000428 learned_addr learned my addr 172.21.15.153:0/2421000428 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient(hunting): get_auth_request con 0x7f09fc0a65a0 auth_method 0 2024-09-10T20:26:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_done global_id 4314 payload 274 2024-09-10T20:26:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 1 -- 172.21.15.153:0/2421000428 >> v1:172.21.15.176:6789/0 conn(0x7f09fc0a5ab0 legacy=0x7f09fc0a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 1 -- 172.21.15.153:0/2421000428 >> v1:172.21.15.153:6789/0 conn(0x7f09fc13e1b0 legacy=0x7f09fc1405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:27.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 1 monclient: found mon.noname-a 2024-09-10T20:26:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:27.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.301+0000 7f09fb7fe640 1 -- 172.21.15.153:0/2421000428 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09fc0a26e0 con 0x7f09fc0a65a0 2024-09-10T20:26:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09fb7fe640 1 -- 172.21.15.153:0/2421000428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc0a65a0 msgr2=0x7f09fc13dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09fb7fe640 1 --2- 172.21.15.153:0/2421000428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc0a65a0 0x7f09fc13dc70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f09f00097c0 tx=0x7f09f001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f967ab8ba4c40c6f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 1 -- 172.21.15.153:0/2421000428 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f00b0070 con 0x7f09fc0a65a0 2024-09-10T20:26:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 10 monclient: dump: 2024-09-10T20:26:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.329 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 1 -- 172.21.15.153:0/2421000428 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09f00ab600 con 0x7f09fc0a65a0 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 1 -- 172.21.15.153:0/2421000428 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f00ab9d0 con 0x7f09fc0a65a0 2024-09-10T20:26:27.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 5 monclient: authenticate success, global_id 4314 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 4 set_mon_vals no callback set 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f09faffd640 10 monclient: dump: 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.331 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 10 monclient: shutdown 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 -- 172.21.15.153:0/2421000428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc0a65a0 msgr2=0x7f09fc13dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 --2- 172.21.15.153:0/2421000428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc0a65a0 0x7f09fc13dc70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f09f00097c0 tx=0x7f09f001c360 comp rx=0 tx=0).stop 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 -- 172.21.15.153:0/2421000428 shutdown_connections 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 --2- 172.21.15.153:0/2421000428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc0a65a0 0x7f09fc13dc70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 -- 172.21.15.153:0/2421000428 >> 172.21.15.153:0/2421000428 conn(0x7f09fc0aba00 msgr2=0x7f09fc0a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 -- 172.21.15.153:0/2421000428 shutdown_connections 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 -- 172.21.15.153:0/2421000428 wait complete. 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 10 monclient: build_initial_monmap 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 10 monclient: monmap: 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:27.332 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 librados: starting msgr at 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 librados: starting objecter 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 Processor -- start 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 -- start start 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 librados: setting wanted keys 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 1 librados: calling monclient init 2024-09-10T20:26:27.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.305+0000 7f0a02401640 10 monclient: init 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09fc0a5ab0 0x7f09fc0a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient(hunting): picked mon.noname-c con 0x7f09fc0a5ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09fc0a65a0 0x7f09fc1385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient(hunting): picked mon.noname-b con 0x7f09fc0a65a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc13e1b0 0x7f09fc1d0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient(hunting): picked mon.noname-a con 0x7f09fc13e1b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09fc0a3b70 con 0x7f09fc13e1b0 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f09fc1444e0 con 0x7f09fc0a65a0 2024-09-10T20:26:27.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f09fc0a90b0 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient(hunting): authenticate will time out at 947.452388s 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09fc0a5ab0 0x7f09fc0a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09fc0a5ab0 0x7f09fc0a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55858/0 (socket says 172.21.15.153:55858) 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 -- 172.21.15.153:0/1980071623 learned_addr learned my addr 172.21.15.153:0/1980071623 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient(hunting): get_auth_request con 0x7f09fc0a5ab0 auth_method 0 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a0099e640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc13e1b0 0x7f09fc1d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a0099e640 10 monclient(hunting): get_auth_request con 0x7f09fc13e1b0 auth_method 0 2024-09-10T20:26:27.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a0099e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a0099e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a0099e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a0099e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a0099e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a0099e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient(hunting): handle_auth_done global_id 4316 payload 824 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fb7fe640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09fc0a65a0 0x7f09fc1385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09fc0a65a0 msgr2=0x7f09fc1385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f09fc0a65a0 0x7f09fc1385b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc13e1b0 msgr2=0x7f09fc1d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc13e1b0 0x7f09fc1d0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 monclient: found mon.noname-c 2024-09-10T20:26:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:26:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 -- 172.21.15.153:0/1980071623 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09fc09f9b0 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a0099e640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc13e1b0 0x7f09fc1d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09fc0a5ab0 msgr2=0x7f09fc0a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fbfff640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09fc0a5ab0 0x7f09fc0a0b00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00c450 tx=0x7f09ec00c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 client.?.objecter ms_handle_connect 0x7f09fc0a5ab0 2024-09-10T20:26:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec012a40 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 monclient: dump: 2024-09-10T20:26:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.339 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.340 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 monclient: _finish_auth 0 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 monclient: _check_auth_tickets 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09ec017070 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 5 monclient: authenticate success, global_id 4316 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient: _renew_subs 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09fc121c50 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09ec012e40 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09f8ff9640 10 monclient: dump: 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient: _renew_subs 2024-09-10T20:26:27.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fa7fc640 4 set_mon_vals no callback set 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f09fc0a1490 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 librados: init done 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc09e9f0 0x7f09fc1d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09fc0a20e0 con 0x7f09fc09e9f0 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.309+0000 7f09fb7fe640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc09e9f0 0x7f09fc1d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 10 monclient: get_auth_request con 0x7f09fc09e9f0 auth_method 0 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f09ec01c460 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_done global_id 4320 payload 824 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 1 -- 172.21.15.153:0/1980071623 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09fc0a5e90 con 0x7f09fc09e9f0 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 1 --2- 172.21.15.153:0/1980071623 >> v2:172.21.15.176:6812/3003473398 conn(0x7f09e803cea0 0x7f09e803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f09ec04f030 con 0x7f09fc0a5ab0 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 10 client.4316.objecter ms_dispatch 0x7f09fc000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 3 client.4316.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 3 client.4316.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 20 client.4316.objecter dump_active .. 0 homeless 2024-09-10T20:26:27.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f0a0099e640 1 --2- 172.21.15.153:0/1980071623 >> v2:172.21.15.176:6812/3003473398 conn(0x7f09e803cea0 0x7f09e803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f0a0099e640 10 monclient: get_auth_request con 0x7f09e803cea0 auth_method 0 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 10 client.4316.objecter ms_handle_connect 0x7f09fc09e9f0 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 10 client.4316.objecter resend_mon_ops 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc09e9f0 msgr2=0x7f09fc1d0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09fb7fe640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc09e9f0 0x7f09fc1d0c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f09f00ad000 tx=0x7f09f001aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f00bb020 con 0x7f09fc09e9f0 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 10 monclient: dump: 2024-09-10T20:26:27.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f09f00b1040 con 0x7f09fc09e9f0 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:27.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc09e9f0 msgr2=0x7f09fc1d0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc09e9f0 0x7f09fc1d0c70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f09f00ad000 tx=0x7f09f001aba0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc09e9f0 msgr2=0x7f09fc1d0c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc09e9f0 0x7f09fc1d0c70 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f09f00ad000 tx=0x7f09f001aba0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f09f8ff9640 10 client.4316.objecter ms_handle_connect 0x7f09e803cea0 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.313+0000 7f0a0099e640 1 --2- 172.21.15.153:0/1980071623 >> v2:172.21.15.176:6812/3003473398 conn(0x7f09e803cea0 0x7f09e803f360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09f4005c10 tx=0x7f09f4009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f0a02401640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f0a02401640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc1d11b0 0x7f09fc13bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f0a02401640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f09fc1d1590 con 0x7f09fc1d11b0 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc1d11b0 0x7f09fc13bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 10 monclient: get_auth_request con 0x7f09fc1d11b0 auth_method 0 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 10 monclient(hunting): handle_auth_done global_id 4323 payload 824 2024-09-10T20:26:27.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 1 -- 172.21.15.153:0/1980071623 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f09fc0a6be0 con 0x7f09fc1d11b0 2024-09-10T20:26:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 10 client.4316.objecter ms_handle_connect 0x7f09fc1d11b0 2024-09-10T20:26:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc1d11b0 msgr2=0x7f09fc13bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 10 client.4316.objecter resend_mon_ops 2024-09-10T20:26:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09fb7fe640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc1d11b0 0x7f09fc13bc60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f09f00097c0 tx=0x7f09f0019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09f00bb070 con 0x7f09fc1d11b0 2024-09-10T20:26:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 10 monclient: dump: 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4721 (secure 0 0 0) 0x7f09f00c22c0 con 0x7f09fc1d11b0 2024-09-10T20:26:27.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:26:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc1d11b0 msgr2=0x7f09fc13bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc1d11b0 0x7f09fc13bc60 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f09f00097c0 tx=0x7f09f0019100 comp rx=0 tx=0).stop 2024-09-10T20:26:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc1d11b0 msgr2=0x7f09fc13bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:27.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.321+0000 7f09f8ff9640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f09fc1d11b0 0x7f09fc13bc60 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f09f00097c0 tx=0x7f09f0019100 comp rx=0 tx=0).stop 2024-09-10T20:26:27.351 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:26:27.351 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:26:27.351 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 8, 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 34011, 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 27, 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:26:27.352 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:26:27.353 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:26:27.354 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:26:27.355 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:26:27.356 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:26:27.357 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:27.358 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:27.359 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:26:27.360 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.361 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:26:27.362 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:26:27.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:26:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.364 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:26:27.364 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:26:27.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 10 librados: watch_flush enter 2024-09-10T20:26:27.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 10 librados: watch_flush exit 2024-09-10T20:26:27.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 20 client.4316.objecter shutdown clearing up homeless session... 2024-09-10T20:26:27.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 10 client.4316.objecter successfully canceled tick 2024-09-10T20:26:27.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 >> v2:172.21.15.176:6812/3003473398 conn(0x7f09e803cea0 msgr2=0x7f09e803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 1 --2- 172.21.15.153:0/1980071623 >> v2:172.21.15.176:6812/3003473398 conn(0x7f09e803cea0 0x7f09e803f360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f09f4005c10 tx=0x7f09f4009040 comp rx=0 tx=0).stop 2024-09-10T20:26:27.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 10 monclient: shutdown 2024-09-10T20:26:27.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:27.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09fc0a5ab0 msgr2=0x7f09fc0a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 1 --2- 172.21.15.153:0/1980071623 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f09fc0a5ab0 0x7f09fc0a0b00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f09ec00c450 tx=0x7f09ec00c910 comp rx=0 tx=0).stop 2024-09-10T20:26:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a0099e640 1 -- 172.21.15.153:0/1980071623 reap_dead start 2024-09-10T20:26:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a0099e640 1 -- 172.21.15.153:0/1980071623 reap_dead start 2024-09-10T20:26:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 shutdown_connections 2024-09-10T20:26:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 >> 172.21.15.153:0/1980071623 conn(0x7f09fc0aba00 msgr2=0x7f09fc0aa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:27.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 shutdown_connections 2024-09-10T20:26:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 1 -- 172.21.15.153:0/1980071623 wait complete. 2024-09-10T20:26:27.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.325+0000 7f0a02401640 1 librados: shutdown 2024-09-10T20:26:27.369 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:26:27.370 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T20:26:27.370 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.465+0000 7ff826a1c640 10 monclient: get_monmap_and_config 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.465+0000 7ff826a1c640 10 monclient: build_initial_monmap 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.465+0000 7ff826a1c640 10 monclient: monmap: 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:27.473581+0000 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:27.473581+0000 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.474 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:27.475 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:27.475 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 1 Processor -- start 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 1 -- start start 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient: init 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff8200a5ab0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff8200a65a0 0x7ff82013dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff8200a65a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff82013e1b0 0x7ff8201405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff82013e1b0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff820070780 con 0x7ff8200a5ab0 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7ff8200a3010 con 0x7ff8200a65a0 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff8200709c0 con 0x7ff82013e1b0 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff826a1c640 10 monclient(hunting): authenticate will time out at 947.612995s 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff8200a5ab0 0x7ff8200a5e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60566/0 (socket says 172.21.15.153:60566) 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81ffff640 1 -- 172.21.15.153:0/3355070920 learned_addr learned my addr 172.21.15.153:0/3355070920 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 1 --2- 172.21.15.153:0/3355070920 >> v2:172.21.15.153:3300/0 conn(0x7ff8200a65a0 0x7ff82013dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request con 0x7ff8200a65a0 auth_method 0 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 1 -- 172.21.15.153:0/3355070920 >> v2:172.21.15.176:3300/0 conn(0x7ff82013e1b0 msgr2=0x7ff8201405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff824fb9640 1 --2- 172.21.15.153:0/3355070920 >> v2:172.21.15.176:3300/0 conn(0x7ff82013e1b0 0x7ff8201405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 1 --2- 172.21.15.153:0/3355070920 >> v2:172.21.15.176:3300/0 conn(0x7ff82013e1b0 0x7ff8201405a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 1 -- 172.21.15.153:0/3355070920 >> v1:172.21.15.153:6789/0 conn(0x7ff8200a5ab0 legacy=0x7ff8200a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 1 monclient: found mon.noname-a 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:27.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.469+0000 7ff81f7fe640 1 -- 172.21.15.153:0/3355070920 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8200a26e0 con 0x7ff8200a65a0 2024-09-10T20:26:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81f7fe640 1 -- 172.21.15.153:0/3355070920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 msgr2=0x7ff82013dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81f7fe640 1 --2- 172.21.15.153:0/3355070920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 0x7ff82013dc70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff8200a3430 tx=0x7ff81001c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=296ee3d7b8a9a96 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 1 -- 172.21.15.153:0/3355070920 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100b0070 con 0x7ff8200a65a0 2024-09-10T20:26:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 10 monclient: dump: 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 10 monclient: _finish_auth 0 2024-09-10T20:26:27.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 1 -- 172.21.15.153:0/3355070920 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8100ab600 con 0x7ff8200a65a0 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 5 monclient: authenticate success, global_id 4326 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 1 -- 172.21.15.153:0/3355070920 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100ab9d0 con 0x7ff8200a65a0 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff81effd640 10 monclient: dump: 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.483 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 4 set_mon_vals no callback set 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 10 monclient: shutdown 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 -- 172.21.15.153:0/3355070920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 msgr2=0x7ff82013dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 --2- 172.21.15.153:0/3355070920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 0x7ff82013dc70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7ff8200a3430 tx=0x7ff81001c340 comp rx=0 tx=0).stop 2024-09-10T20:26:27.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 -- 172.21.15.153:0/3355070920 shutdown_connections 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 --2- 172.21.15.153:0/3355070920 >> v2:172.21.15.176:3300/0 conn(0x7ff82013e1b0 0x7ff8201405a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 --2- 172.21.15.153:0/3355070920 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 0x7ff82013dc70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 -- 172.21.15.153:0/3355070920 >> 172.21.15.153:0/3355070920 conn(0x7ff8200aba00 msgr2=0x7ff8200a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 -- 172.21.15.153:0/3355070920 shutdown_connections 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 -- 172.21.15.153:0/3355070920 wait complete. 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 10 monclient: build_initial_monmap 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 10 monclient: monmap: 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:27.485 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:27.486 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:27.486 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 librados: starting msgr at 2024-09-10T20:26:27.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 librados: starting objecter 2024-09-10T20:26:27.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 Processor -- start 2024-09-10T20:26:27.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 -- start start 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 librados: setting wanted keys 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 1 librados: calling monclient init 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.473+0000 7ff826a1c640 10 monclient: init 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8200a5ab0 0x7ff8201380c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient(hunting): picked mon.noname-b con 0x7ff8200a5ab0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 0x7ff82009f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient(hunting): picked mon.noname-a con 0x7ff8200a65a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff82013e1b0 0x7ff82009f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient(hunting): picked mon.noname-c con 0x7ff82013e1b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff8200a3b70 con 0x7ff8200a65a0 2024-09-10T20:26:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff820143300 con 0x7ff8200a5ab0 2024-09-10T20:26:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff8201394b0 con 0x7ff82013e1b0 2024-09-10T20:26:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient(hunting): authenticate will time out at 947.620428s 2024-09-10T20:26:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 0x7ff82009f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff824fb9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff82013e1b0 0x7ff82009f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 0x7ff82009f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42446/0 (socket says 172.21.15.153:42446) 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 -- 172.21.15.153:0/2520504956 learned_addr learned my addr 172.21.15.153:0/2520504956 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request con 0x7ff8200a65a0 auth_method 0 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff824fb9640 10 monclient(hunting): get_auth_request con 0x7ff82013e1b0 auth_method 0 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff824fb9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff824fb9640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff824fb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff824fb9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff824fb9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff824fb9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81ffff640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8200a5ab0 0x7ff8201380c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81ffff640 10 monclient(hunting): get_auth_request con 0x7ff8200a5ab0 auth_method 0 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff82013e1b0 msgr2=0x7ff82009f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff82013e1b0 0x7ff82009f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8200a5ab0 msgr2=0x7ff8201380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff8200a5ab0 0x7ff8201380c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 monclient: found mon.noname-a 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 -- 172.21.15.153:0/2520504956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff820121720 con 0x7ff8200a65a0 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 msgr2=0x7ff82009f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81f7fe640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 0x7ff82009f240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8100ae040 tx=0x7ff81001ab80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 client.?.objecter ms_handle_connect 0x7ff8200a65a0 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100b0050 con 0x7ff8200a65a0 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 monclient: dump: 2024-09-10T20:26:27.496 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 monclient: _finish_auth 0 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 monclient: _check_auth_tickets 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8100bc070 con 0x7ff8200a65a0 2024-09-10T20:26:27.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8100b5c80 con 0x7ff8200a65a0 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81cff9640 10 monclient: dump: 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff81e7fc640 4 set_mon_vals no callback set 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 5 monclient: authenticate success, global_id 4329 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient: _renew_subs 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff820122b50 con 0x7ff8200a65a0 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 client.4329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient: _renew_subs 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff820122870 con 0x7ff8200a65a0 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.477+0000 7ff826a1c640 1 librados: init done 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff826a1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff826a1c640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff820138600 0x7ff82009fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff826a1c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff8200a20e0 con 0x7ff820138600 2024-09-10T20:26:27.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff820138600 0x7ff82009fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 10 monclient: get_auth_request con 0x7ff820138600 auth_method 0 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7ff8100ab4c0 con 0x7ff8200a65a0 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 10 monclient(hunting): handle_auth_done global_id 4322 payload 824 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 1 -- 172.21.15.153:0/2520504956 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff8200a2300 con 0x7ff820138600 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 1 --2- 172.21.15.153:0/2520504956 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff80c03cf20 0x7ff80c03f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7ff8100e9550 con 0x7ff8200a65a0 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 10 client.4329.objecter ms_dispatch 0x7ff820000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:27.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 3 client.4329.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 3 client.4329.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 20 client.4329.objecter dump_active .. 0 homeless 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff824fb9640 1 --2- 172.21.15.153:0/2520504956 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff80c03cf20 0x7ff80c03f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff824fb9640 10 monclient: get_auth_request con 0x7ff80c03cf20 auth_method 0 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 10 client.4329.objecter ms_handle_connect 0x7ff820138600 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 10 client.4329.objecter resend_mon_ops 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff820138600 msgr2=0x7ff82009fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81ffff640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff820138600 0x7ff82009fcc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff81400e500 tx=0x7ff81400e9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff814004e40 con 0x7ff820138600 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 10 monclient: dump: 2024-09-10T20:26:27.501 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff814018070 con 0x7ff820138600 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:27.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff820138600 msgr2=0x7ff82009fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff820138600 0x7ff82009fcc0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff81400e500 tx=0x7ff81400e9c0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff820138600 msgr2=0x7ff82009fcc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:27.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.481+0000 7ff81cff9640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff820138600 0x7ff82009fcc0 secure :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff81400e500 tx=0x7ff81400e9c0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.485+0000 7ff81cff9640 10 client.4329.objecter ms_handle_connect 0x7ff80c03cf20 2024-09-10T20:26:27.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.485+0000 7ff824fb9640 1 --2- 172.21.15.153:0/2520504956 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff80c03cf20 0x7ff80c03f3e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff8200a9310 tx=0x7ff8180076f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff826a1c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff826a1c640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8200a0200 0x7ff82013bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff826a1c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff8200a6be0 con 0x7ff8200a0200 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8200a0200 0x7ff82013bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 10 monclient: get_auth_request con 0x7ff8200a0200 auth_method 0 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 10 monclient(hunting): handle_auth_done global_id 4325 payload 824 2024-09-10T20:26:27.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 1 -- 172.21.15.153:0/2520504956 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff8200a0610 con 0x7ff8200a0200 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 10 client.4329.objecter ms_handle_connect 0x7ff8200a0200 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 10 client.4329.objecter resend_mon_ops 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8200a0200 msgr2=0x7ff82013bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81ffff640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8200a0200 0x7ff82013bc60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff81400e4d0 tx=0x7ff814004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8140064e0 con 0x7ff8200a0200 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 10 monclient: dump: 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.506 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4719 (secure 0 0 0) 0x7ff8140064e0 con 0x7ff8200a0200 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8200a0200 msgr2=0x7ff82013bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8200a0200 0x7ff82013bc60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff81400e4d0 tx=0x7ff814004950 comp rx=0 tx=0).stop 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8200a0200 msgr2=0x7ff82013bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:27.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.489+0000 7ff81cff9640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff8200a0200 0x7ff82013bc60 secure :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff81400e4d0 tx=0x7ff814004950 comp rx=0 tx=0).stop 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 8, 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 34179, 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.508 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 27, 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:27.509 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:27.510 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:27.511 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:27.512 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:27.513 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:26:27.514 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.515 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:27.516 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:27.517 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:26:27.518 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.519 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:26:27.520 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.520 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.520 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.520 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:26:27.520 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.520 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.520 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.520 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:26:27.520 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:26:27.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 10 librados: watch_flush enter 2024-09-10T20:26:27.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 10 librados: watch_flush exit 2024-09-10T20:26:27.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 20 client.4329.objecter shutdown clearing up homeless session... 2024-09-10T20:26:27.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 10 client.4329.objecter successfully canceled tick 2024-09-10T20:26:27.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff80c03cf20 msgr2=0x7ff80c03f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 1 --2- 172.21.15.153:0/2520504956 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff80c03cf20 0x7ff80c03f3e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff8200a9310 tx=0x7ff8180076f0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 10 monclient: shutdown 2024-09-10T20:26:27.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 msgr2=0x7ff82009f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 1 --2- 172.21.15.153:0/2520504956 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff8200a65a0 0x7ff82009f240 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff8100ae040 tx=0x7ff81001ab80 comp rx=0 tx=0).stop 2024-09-10T20:26:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff824fb9640 1 -- 172.21.15.153:0/2520504956 reap_dead start 2024-09-10T20:26:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff824fb9640 1 -- 172.21.15.153:0/2520504956 reap_dead start 2024-09-10T20:26:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 shutdown_connections 2024-09-10T20:26:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 >> 172.21.15.153:0/2520504956 conn(0x7ff8200aba00 msgr2=0x7ff82013ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 shutdown_connections 2024-09-10T20:26:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 1 -- 172.21.15.153:0/2520504956 wait complete. 2024-09-10T20:26:27.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.493+0000 7ff826a1c640 1 librados: shutdown 2024-09-10T20:26:27.523 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:26:27.523 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T20:26:27.524 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:26:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient: get_monmap_and_config 2024-09-10T20:26:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient: build_initial_monmap 2024-09-10T20:26:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient: monmap: 2024-09-10T20:26:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:27.624 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:27.623904+0000 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:27.623904+0000 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:27.625 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 1 Processor -- start 2024-09-10T20:26:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 1 -- start start 2024-09-10T20:26:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient: init 2024-09-10T20:26:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-10T20:26:27.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f80a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient(hunting): picked mon.noname-b con 0x7f12f80a5ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f12f80a63d0 0x7f12f81389c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient(hunting): picked mon.noname-a con 0x7f12f80a63d0 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f12f8138f00 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12f8070780 con 0x7f12f8138f00 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f12f80a3010 con 0x7f12f80a63d0 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f12f80709c0 con 0x7f12f80a5ab0 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12ff10d640 10 monclient(hunting): authenticate will time out at 947.763227s 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f12f80a63d0 0x7f12f81389c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f80a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f80a5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55912/0 (socket says 172.21.15.153:55912) 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 1 -- 172.21.15.153:0/844704586 learned_addr learned my addr 172.21.15.153:0/844704586 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f12f80a63d0 0x7f12f81389c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42460/0 (socket says 172.21.15.153:42460) 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient(hunting): get_auth_request con 0x7f12f80a5ab0 auth_method 0 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 10 monclient(hunting): get_auth_request con 0x7f12f80a63d0 auth_method 0 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient(hunting): handle_auth_done global_id 4328 payload 274 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 1 -- 172.21.15.153:0/844704586 >> v2:172.21.15.153:3300/0 conn(0x7f12f80a63d0 msgr2=0x7f12f81389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 1 --2- 172.21.15.153:0/844704586 >> v2:172.21.15.153:3300/0 conn(0x7f12f80a63d0 0x7f12f81389c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 1 -- 172.21.15.153:0/844704586 >> v1:172.21.15.176:6789/0 conn(0x7f12f8138f00 legacy=0x7f12f813fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient: _finish_hunting 0 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 1 monclient: found mon.noname-b 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:26:27.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12fcea9640 1 -- 172.21.15.153:0/844704586 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f80a26e0 con 0x7f12f80a5ab0 2024-09-10T20:26:27.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.617+0000 7f12effff640 1 --2- 172.21.15.153:0/844704586 >> v2:172.21.15.153:3300/0 conn(0x7f12f80a63d0 0x7f12f81389c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12fcea9640 1 -- 172.21.15.153:0/844704586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 msgr2=0x7f12f80a5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12fcea9640 1 --2- 172.21.15.153:0/844704586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f80a5e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12f4009870 tx=0x7f12f401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=397943912aa1bad0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ef7fe640 1 -- 172.21.15.153:0/844704586 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f40b0070 con 0x7f12f80a5ab0 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ef7fe640 10 monclient: dump: 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ef7fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ef7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ff10d640 5 monclient: authenticate success, global_id 4328 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ef7fe640 1 -- 172.21.15.153:0/844704586 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12f40ab600 con 0x7f12f80a5ab0 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.621+0000 7f12ff10d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ef7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 4 set_mon_vals no callback set 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ef7fe640 1 -- 172.21.15.153:0/844704586 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f40ab9d0 con 0x7f12f80a5ab0 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ef7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ef7fe640 10 monclient: dump: 2024-09-10T20:26:27.638 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient: shutdown 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 -- 172.21.15.153:0/844704586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 msgr2=0x7f12f80a5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 --2- 172.21.15.153:0/844704586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f80a5e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f12f4009870 tx=0x7f12f401c420 comp rx=0 tx=0).stop 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 -- 172.21.15.153:0/844704586 shutdown_connections 2024-09-10T20:26:27.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 --2- 172.21.15.153:0/844704586 >> v2:172.21.15.153:3300/0 conn(0x7f12f80a63d0 0x7f12f81389c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 --2- 172.21.15.153:0/844704586 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f80a5e90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 -- 172.21.15.153:0/844704586 >> 172.21.15.153:0/844704586 conn(0x7f12f80aba00 msgr2=0x7f12f80a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 -- 172.21.15.153:0/844704586 shutdown_connections 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 -- 172.21.15.153:0/844704586 wait complete. 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient: build_initial_monmap 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient: monmap: 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.640 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 librados: starting msgr at 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 librados: starting objecter 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 Processor -- start 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 -- start start 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 librados: setting wanted keys 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 librados: calling monclient init 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient: init 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f8133a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient(hunting): picked mon.noname-c con 0x7f12f80a5ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12f80a63d0 0x7f12f8133fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient(hunting): picked mon.noname-a con 0x7f12f80a63d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f8138f00 0x7f12f8139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient(hunting): picked mon.noname-b con 0x7f12f8138f00 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f12f80a2050 con 0x7f12f80a63d0 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f12f80a1ed0 con 0x7f12f8138f00 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f12f80a3960 con 0x7f12f80a5ab0 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12ff10d640 10 monclient(hunting): authenticate will time out at 947.770009s 2024-09-10T20:26:27.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f8133a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f8133a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:55916/0 (socket says 172.21.15.153:55916) 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 1 -- 172.21.15.153:0/2236138141 learned_addr learned my addr 172.21.15.153:0/2236138141 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 10 monclient(hunting): get_auth_request con 0x7f12f80a5ab0 auth_method 0 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12f80a63d0 0x7f12f8133fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient(hunting): get_auth_request con 0x7f12f80a63d0 auth_method 0 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient(hunting): handle_auth_done global_id 4335 payload 824 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fd6aa640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f8138f00 0x7f12f8139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 msgr2=0x7f12f8133a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f8133a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f8138f00 msgr2=0x7f12f8139c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f8138f00 0x7f12f8139c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 1 monclient: found mon.noname-a 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:27.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12effff640 1 -- 172.21.15.153:0/2236138141 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12f813d210 con 0x7f12f80a63d0 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.625+0000 7f12fcea9640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f12f80a5ab0 0x7f12f8133a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12effff640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12f80a63d0 msgr2=0x7f12f8133fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12effff640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12f80a63d0 0x7f12f8133fc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f12e800e9b0 tx=0x7f12e800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 client.?.objecter ms_handle_connect 0x7f12f80a63d0 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8017070 con 0x7f12f80a63d0 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 monclient: dump: 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.645 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 monclient: _finish_auth 0 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 monclient: _check_auth_tickets 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 5 monclient: authenticate success, global_id 4335 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f12e800cbb0 con 0x7f12f80a63d0 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 10 monclient: _renew_subs 2024-09-10T20:26:27.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12f813d4f0 con 0x7f12f80a63d0 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e8013470 con 0x7f12f80a63d0 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 monclient: dump: 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.647 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 10 monclient: _renew_subs 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12f80a9c20 con 0x7f12f80a63d0 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12eeffd640 4 set_mon_vals no callback set 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 1 librados: init done 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813a190 0x7f12f813a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f12f80a5e90 con 0x7f12f813a190 2024-09-10T20:26:27.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f12e801a070 con 0x7f12f80a63d0 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813a190 0x7f12f813a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 10 monclient: get_auth_request con 0x7f12f813a190 auth_method 0 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 1 --2- 172.21.15.153:0/2236138141 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12e403cea0 0x7f12e403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f12e804f1e0 con 0x7f12f80a63d0 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 10 client.4335.objecter ms_dispatch 0x7f12f8000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 3 client.4335.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 3 client.4335.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12ed7fa640 20 client.4335.objecter dump_active .. 0 homeless 2024-09-10T20:26:27.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fd6aa640 1 --2- 172.21.15.153:0/2236138141 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12e403cea0 0x7f12e403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fd6aa640 10 monclient: get_auth_request con 0x7f12e403cea0 auth_method 0 2024-09-10T20:26:27.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 10 monclient(hunting): handle_auth_done global_id 4276 payload 824 2024-09-10T20:26:27.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.629+0000 7f12fcea9640 1 -- 172.21.15.153:0/2236138141 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f12f80a6be0 con 0x7f12f813a190 2024-09-10T20:26:27.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 10 client.4335.objecter ms_handle_connect 0x7f12f813a190 2024-09-10T20:26:27.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 10 client.4335.objecter resend_mon_ops 2024-09-10T20:26:27.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12fcea9640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813a190 msgr2=0x7f12f813a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12fcea9640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813a190 0x7f12f813a570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f12f401ccc0 tx=0x7f12f40191c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 10 client.4335.objecter ms_handle_connect 0x7f12e403cea0 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12fd6aa640 1 --2- 172.21.15.153:0/2236138141 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12e403cea0 0x7f12e403f360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12f00043d0 tx=0x7f12f0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f40b9020 con 0x7f12f813a190 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 10 monclient: dump: 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.653 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f12f40b1040 con 0x7f12f813a190 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813a190 msgr2=0x7f12f813a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813a190 0x7f12f813a570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f12f401ccc0 tx=0x7f12f40191c0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813a190 msgr2=0x7f12f813a570 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.633+0000 7f12ed7fa640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813a190 0x7f12f813a570 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.637+0000 7f12ff10d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.637+0000 7f12ff10d640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813aab0 0x7f12f8137c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.637+0000 7f12ff10d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:27.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.637+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f12f806a440 con 0x7f12f813aab0 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813aab0 0x7f12f8137c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 10 monclient: get_auth_request con 0x7f12f813aab0 auth_method 0 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 10 monclient(hunting): handle_auth_done global_id 4279 payload 824 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 1 -- 172.21.15.153:0/2236138141 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f12f813cf30 con 0x7f12f813aab0 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813aab0 msgr2=0x7f12f8137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12fcea9640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813aab0 0x7f12f8137c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f12f40b7000 tx=0x7f12f4009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12ed7fa640 10 client.4335.objecter ms_handle_connect 0x7f12f813aab0 2024-09-10T20:26:27.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12ed7fa640 10 client.4335.objecter resend_mon_ops 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f40b9030 con 0x7f12f813aab0 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12ed7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12ed7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.641+0000 7f12ed7fa640 10 monclient: dump: 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:27.656 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4886 (secure 0 0 0) 0x7f12f40c0e10 con 0x7f12f813aab0 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ed7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ed7fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813aab0 msgr2=0x7f12f8137c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ed7fa640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813aab0 0x7f12f8137c80 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f12f40b7000 tx=0x7f12f4009240 comp rx=0 tx=0).stop 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ed7fa640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813aab0 msgr2=0x7f12f8137c80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ed7fa640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f12f813aab0 0x7f12f8137c80 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f12f40b7000 tx=0x7f12f4009240 comp rx=0 tx=0).stop 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:26:27.657 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 8, 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 34291, 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 28, 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:27.658 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:27.659 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:26:27.660 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:26:27.661 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.662 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.663 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:26:27.664 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.665 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:26:27.666 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:27.667 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.668 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:27.669 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:26:27.670 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:27.670 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:27.670 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:27.670 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:26:27.670 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 10 librados: watch_flush enter 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 10 librados: watch_flush exit 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 20 client.4335.objecter shutdown clearing up homeless session... 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 10 client.4335.objecter successfully canceled tick 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12e403cea0 msgr2=0x7f12e403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 --2- 172.21.15.153:0/2236138141 >> v2:172.21.15.176:6812/3003473398 conn(0x7f12e403cea0 0x7f12e403f360 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f12f00043d0 tx=0x7f12f0009040 comp rx=0 tx=0).stop 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 10 monclient: shutdown 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12f80a63d0 msgr2=0x7f12f8133fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12f80a63d0 0x7f12f8133fc0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f12e800e9b0 tx=0x7f12e800ee70 comp rx=0 tx=0).stop 2024-09-10T20:26:27.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12fd6aa640 1 -- 172.21.15.153:0/2236138141 reap_dead start 2024-09-10T20:26:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 shutdown_connections 2024-09-10T20:26:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 --2- 172.21.15.153:0/2236138141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f12f80a63d0 0x7f12f8133fc0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 >> 172.21.15.153:0/2236138141 conn(0x7f12f80aba00 msgr2=0x7f12f80a8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 shutdown_connections 2024-09-10T20:26:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 -- 172.21.15.153:0/2236138141 wait complete. 2024-09-10T20:26:27.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:27.645+0000 7f12ff10d640 1 librados: shutdown 2024-09-10T20:26:27.673 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:26:27.673 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T20:26:27.674 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-09-10T20:26:27.674 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T20:26:27.674 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T20:26:27.674 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T20:26:27.674 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:26:27.684 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T20:26:27.684 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T20:26:27.685 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T20:26:27.685 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T20:26:42.686 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T20:26:42.687 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T20:26:42.687 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T20:26:42.687 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient: get_monmap_and_config 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient: build_initial_monmap 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient: monmap: 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:42.815520+0000 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:42.815520+0000 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:42.816 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:42.817 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:42.817 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:42.817 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:42.817 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:42.817 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:42.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 1 Processor -- start 2024-09-10T20:26:42.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 1 -- start start 2024-09-10T20:26:42.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient: init 2024-09-10T20:26:42.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:42.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-10T20:26:42.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:42.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient(hunting): picked mon.noname-c con 0x7ff0641364f0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:42.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 0x7ff06409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient(hunting): picked mon.noname-b con 0x7ff0641360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient(hunting): picked mon.noname-a con 0x7ff0640a1990 addr v2:172.21.15.153:3300/0 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7ff0640a27a0 con 0x7ff0640a1990 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff0640a2a80 con 0x7ff0641364f0 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff0640a2d60 con 0x7ff0641360a0 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff069f27640 10 monclient(hunting): authenticate will time out at 962.955369s 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff05affd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 msgr2=0x7ff06409ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff05affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 0x7ff06409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:42.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40718/0 (socket says 172.21.15.153:40718) 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 learned_addr learned my addr 172.21.15.153:0/1347312436 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 10 monclient(hunting): get_auth_request con 0x7ff0640a1990 auth_method 0 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 msgr2=0x7ff06409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 msgr2=0x7ff06409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff0637fe640 10 monclient(hunting): get_auth_request con 0x7ff0641364f0 auth_method 0 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:42.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff0637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:42.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff0637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:42.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff0637fe640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 msgr2=0x7ff0641368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:26:42.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff0637fe640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 msgr2=0x7ff0641368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:26:42.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.813+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:42.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:42.817+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff063fff640 10 monclient(hunting): get_auth_request con 0x7ff0640a1990 auth_method 0 2024-09-10T20:26:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff063fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff063fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 msgr2=0x7ff06409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:26:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 msgr2=0x7ff06409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:26:43.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:43.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:26:43.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff05affd640 1 -- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 msgr2=0x7ff06409ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:26:43.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.017+0000 7ff05affd640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 0x7ff06409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:26:43.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.021+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:43.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.021+0000 7ff0637fe640 10 monclient(hunting): get_auth_request con 0x7ff0641364f0 auth_method 0 2024-09-10T20:26:43.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.021+0000 7ff0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:43.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.021+0000 7ff0637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:43.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.021+0000 7ff0637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:43.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.021+0000 7ff0637fe640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 msgr2=0x7ff0641368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:26:43.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.021+0000 7ff0637fe640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 msgr2=0x7ff0641368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:26:43.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.021+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:43.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.021+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:26:43.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff05affd640 1 -- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 msgr2=0x7ff06409ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:26:43.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:43.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff05affd640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 0x7ff06409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:26:43.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff063fff640 10 monclient(hunting): get_auth_request con 0x7ff0640a1990 auth_method 0 2024-09-10T20:26:43.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:43.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff063fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:43.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff063fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:43.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 msgr2=0x7ff06409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:26:43.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 msgr2=0x7ff06409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:26:43.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:43.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.421+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:26:43.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.425+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:43.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.425+0000 7ff0637fe640 10 monclient(hunting): get_auth_request con 0x7ff0641364f0 auth_method 0 2024-09-10T20:26:43.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.425+0000 7ff0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:43.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.425+0000 7ff0637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:43.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.425+0000 7ff0637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:43.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.425+0000 7ff0637fe640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 msgr2=0x7ff0641368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:26:43.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.425+0000 7ff0637fe640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 msgr2=0x7ff0641368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:26:43.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.425+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:43.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:43.425+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:26:44.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff05affd640 1 -- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 msgr2=0x7ff06409ce10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:26:44.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff05affd640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 0x7ff06409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:26:44.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:44.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff063fff640 10 monclient(hunting): get_auth_request con 0x7ff0640a1990 auth_method 0 2024-09-10T20:26:44.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:44.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff063fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:44.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff063fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:44.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 msgr2=0x7ff06409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:26:44.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 msgr2=0x7ff06409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:26:44.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:44.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.225+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:26:44.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.229+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:44.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.229+0000 7ff0637fe640 10 monclient(hunting): get_auth_request con 0x7ff0641364f0 auth_method 0 2024-09-10T20:26:44.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.229+0000 7ff0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:44.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.229+0000 7ff0637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:44.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.229+0000 7ff0637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:44.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.229+0000 7ff0637fe640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 msgr2=0x7ff0641368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:26:44.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.229+0000 7ff0637fe640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 msgr2=0x7ff0641368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:26:44.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.229+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:26:44.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:44.229+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:26:45.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 10 monclient(hunting): tick 2024-09-10T20:26:45.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-10T20:26:45.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 1 monclient(hunting): continuing hunt 2024-09-10T20:26:45.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T20:26:45.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 1 -- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 msgr2=0x7ff06409ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:26:45.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff0641360a0 0x7ff06409ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:45.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 msgr2=0x7ff0641368d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:26:45.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.176:3300/0 conn(0x7ff0641364f0 0x7ff0641368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:45.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 1 -- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 msgr2=0x7ff06409d350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:26:45.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 1 --2- 172.21.15.153:0/1347312436 >> v2:172.21.15.153:3300/0 conn(0x7ff0640a1990 0x7ff06409d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:45.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 10 monclient: _add_conns ranks=[0,1] 2024-09-10T20:26:45.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff0440036c0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:45.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff044007410 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:45.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 1 -- 172.21.15.153:0/1347312436 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff064136e10 con 0x7ff0440036c0 2024-09-10T20:26:45.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 1 -- 172.21.15.153:0/1347312436 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff04400c150 con 0x7ff044007410 2024-09-10T20:26:45.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.813+0000 7ff061ffb640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:45.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.817+0000 7ff0637fe640 1 -- 172.21.15.153:0/1347312436 >> v1:172.21.15.153:6789/0 conn(0x7ff0440036c0 legacy=0x7ff044005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:45.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:45.817+0000 7ff05affd640 1 -- 172.21.15.153:0/1347312436 >> v1:172.21.15.176:6789/0 conn(0x7ff044007410 legacy=0x7ff0440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:47.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.741+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 900071422 0 0) 0x7ff064136e10 con 0x7ff0440036c0 2024-09-10T20:26:47.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.741+0000 7ff062ffd640 10 monclient(hunting): _negotiate 2024-09-10T20:26:47.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.741+0000 7ff062ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:47.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.741+0000 7ff062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:47.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.741+0000 7ff062ffd640 10 monclient(hunting): my global_id is 4338 2024-09-10T20:26:47.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.741+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff050003530 con 0x7ff0440036c0 2024-09-10T20:26:47.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.745+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2693824049 0 0) 0x7ff04400c150 con 0x7ff044007410 2024-09-10T20:26:47.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.745+0000 7ff062ffd640 10 monclient(hunting): _negotiate 2024-09-10T20:26:47.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.745+0000 7ff062ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:47.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.745+0000 7ff062ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:47.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.745+0000 7ff062ffd640 10 monclient(hunting): my global_id is 4282 2024-09-10T20:26:47.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.745+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff064136e10 con 0x7ff044007410 2024-09-10T20:26:47.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1507006722 0 0) 0x7ff050003530 con 0x7ff0440036c0 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 >> v1:172.21.15.176:6789/0 conn(0x7ff044007410 legacy=0x7ff0440098d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 monclient: found mon.noname-a-legacy 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff04400d330 con 0x7ff0440036c0 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3765011595 0 0) 0x7ff05400f040 con 0x7ff0440036c0 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T20:26:47.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient: dump: 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 >> v1:172.21.15.153:6789/0 conn(0x7ff0440036c0 legacy=0x7ff044005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:47.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 reap_dead start 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff050003bd0 0x7ff050003fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient(hunting): picked mon.a con 0x7ff050003bd0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 0x7ff050004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient(hunting): picked mon.c con 0x7ff050004810 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0500099c0 0x7ff05000bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient(hunting): picked mon.b con 0x7ff0500099c0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff05000c320 con 0x7ff050003bd0 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff05000c4d0 con 0x7ff0500099c0 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:47.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff05000c6d0 con 0x7ff050004810 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff05affd640 1 -- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 msgr2=0x7ff050004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 10 monclient(hunting): _finish_auth 0 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 DispatchQueue::entry inject delay of 0.002000 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff05affd640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 0x7ff050004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0500099c0 0x7ff05000bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient(hunting): get_auth_request con 0x7ff0500099c0 auth_method 0 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff050003bd0 0x7ff050003fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff0637fe640 10 monclient(hunting): get_auth_request con 0x7ff050003bd0 auth_method 0 2024-09-10T20:26:47.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff0637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff0637fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff0637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff0637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff0637fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient(hunting): handle_auth_done global_id 4338 payload 293 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 msgr2=0x7ff050004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 0x7ff050004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff050003bd0 msgr2=0x7ff050003fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff050003bd0 0x7ff050003fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:47.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:47.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 1 monclient: found mon.b 2024-09-10T20:26:47.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:47.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:47.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff050003530 con 0x7ff0500099c0 2024-09-10T20:26:47.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient: _finish_auth 0 2024-09-10T20:26:47.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 10 monclient: _check_auth_tickets 2024-09-10T20:26:47.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff063fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:47.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff069f27640 5 monclient: authenticate success, global_id 4338 2024-09-10T20:26:47.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.749+0000 7ff069f27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:47.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff063fff640 1 -- 172.21.15.153:0/1347312436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0500099c0 msgr2=0x7ff05000bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff063fff640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0500099c0 0x7ff05000bdb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff0641306c0 tx=0x7ff05c01d110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ecff34f719eb6cc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c0b3070 con 0x7ff0500099c0 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 10 monclient: dump: 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:47.761 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff05c0ae410 con 0x7ff0500099c0 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 4 set_mon_vals no callback set 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 1 -- 172.21.15.153:0/1347312436 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05c009700 con 0x7ff0500099c0 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff062ffd640 10 monclient: dump: 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:47.762 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 10 monclient: shutdown 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 -- 172.21.15.153:0/1347312436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0500099c0 msgr2=0x7ff05000bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0500099c0 0x7ff05000bdb0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff0641306c0 tx=0x7ff05c01d110 comp rx=0 tx=0).stop 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 -- 172.21.15.153:0/1347312436 shutdown_connections 2024-09-10T20:26:47.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff0500099c0 0x7ff05000bdb0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:47.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 0x7ff050004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:47.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 --2- 172.21.15.153:0/1347312436 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff050003bd0 0x7ff050003fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:47.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 -- 172.21.15.153:0/1347312436 >> 172.21.15.153:0/1347312436 conn(0x7ff0640a71b0 msgr2=0x7ff0640a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:47.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 -- 172.21.15.153:0/1347312436 shutdown_connections 2024-09-10T20:26:47.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 -- 172.21.15.153:0/1347312436 wait complete. 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 10 monclient: build_initial_monmap 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 10 monclient: monmap: 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:47.768 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 librados: starting msgr at 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 librados: starting objecter 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 Processor -- start 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 -- start start 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 librados: setting wanted keys 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 1 librados: calling monclient init 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.757+0000 7ff069f27640 10 monclient: init 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff050003bd0 0x7ff06412f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient(hunting): picked mon.noname-b con 0x7ff050003bd0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 0x7ff06412fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:47.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient(hunting): picked mon.noname-c con 0x7ff050004810 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0500099c0 0x7ff0641bffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient(hunting): picked mon.noname-a con 0x7ff0500099c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff06409d1a0 con 0x7ff0500099c0 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff06409d020 con 0x7ff050003bd0 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff06409d320 con 0x7ff050004810 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient(hunting): authenticate will time out at 967.900585s 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff05affd640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 msgr2=0x7ff06412fa60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff05affd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 0x7ff06412fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:47.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff050003bd0 0x7ff06412f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff063fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0500099c0 0x7ff0641bffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff050003bd0 0x7ff06412f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.153:50250/0 (socket says 172.21.15.153:50250) 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 -- 172.21.15.153:0/1648243423 learned_addr learned my addr 172.21.15.153:0/1648243423 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient(hunting): get_auth_request con 0x7ff050003bd0 auth_method 0 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff063fff640 10 monclient(hunting): get_auth_request con 0x7ff0500099c0 auth_method 0 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff063fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff063fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff063fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:47.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff063fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff063fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff063fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient(hunting): handle_auth_done global_id 4285 payload 824 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 -- 172.21.15.153:0/1648243423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 msgr2=0x7ff06412fa60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1648243423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 0x7ff06412fa60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 -- 172.21.15.153:0/1648243423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0500099c0 msgr2=0x7ff0641bffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1648243423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0500099c0 0x7ff0641bffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient: _finish_hunting 0 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 monclient: found mon.noname-b 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 -- 172.21.15.153:0/1648243423 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0641c3b10 con 0x7ff050003bd0 2024-09-10T20:26:47.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff063fff640 1 --2- 172.21.15.153:0/1648243423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0500099c0 0x7ff0641bffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 -- 172.21.15.153:0/1648243423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff050003bd0 msgr2=0x7ff06412f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff0637fe640 1 --2- 172.21.15.153:0/1648243423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff050003bd0 0x7ff06412f520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff05400e620 tx=0x7ff0540146c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 client.?.objecter ms_handle_connect 0x7ff050003bd0 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 1 -- 172.21.15.153:0/1648243423 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff054014e30 con 0x7ff050003bd0 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 monclient: dump: 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.773 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 monclient: _finish_auth 0 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 monclient: _check_auth_tickets 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 1 -- 172.21.15.153:0/1648243423 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff054003620 con 0x7ff050003bd0 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 5 monclient: authenticate success, global_id 4285 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient: _renew_subs 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:47.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0641c3df0 con 0x7ff050003bd0 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 1 -- 172.21.15.153:0/1648243423 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0540039f0 con 0x7ff050003bd0 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff060ff9640 10 monclient: dump: 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:47.775 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient: _renew_subs 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0641c1a90 con 0x7ff050003bd0 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff069f27640 1 librados: init done 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.761+0000 7ff061ffb640 4 set_mon_vals no callback set 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff069f27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff069f27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff069f27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0640a27a0 con 0x7ff050003bd0 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff060ff9640 1 -- 172.21.15.153:0/1648243423 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7ff054003b90 con 0x7ff050003bd0 2024-09-10T20:26:47.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff060ff9640 1 --2- 172.21.15.153:0/1648243423 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff04403d7a0 0x7ff04403fc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:47.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff060ff9640 1 -- 172.21.15.153:0/1648243423 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7ff0540503b0 con 0x7ff050003bd0 2024-09-10T20:26:47.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff060ff9640 10 client.4285.objecter ms_dispatch 0x7ff064000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:47.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff060ff9640 3 client.4285.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:47.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff060ff9640 3 client.4285.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:47.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff060ff9640 20 client.4285.objecter dump_active .. 0 homeless 2024-09-10T20:26:47.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff05affd640 1 --2- 172.21.15.153:0/1648243423 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff04403d7a0 0x7ff04403fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:47.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.765+0000 7ff05affd640 10 monclient: get_auth_request con 0x7ff04403d7a0 auth_method 0 2024-09-10T20:26:47.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.769+0000 7ff05affd640 1 --2- 172.21.15.153:0/1648243423 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff04403d7a0 0x7ff04403fc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff04c003cc0 tx=0x7ff04c003bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:47.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.769+0000 7ff060ff9640 10 client.4285.objecter ms_handle_connect 0x7ff04403d7a0 2024-09-10T20:26:47.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.773+0000 7ff060ff9640 1 -- 172.21.15.153:0/1648243423 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7ff054017d40 con 0x7ff050003bd0 2024-09-10T20:26:47.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.773+0000 7ff060ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:47.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:47.773+0000 7ff060ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:48.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.001+0000 7ff069f27640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:26:48.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.001+0000 7ff069f27640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:26:48.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.001+0000 7ff069f27640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:26:48.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.001+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff0641c3550 con 0x7ff050003bd0 2024-09-10T20:26:48.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff060ff9640 1 -- 172.21.15.153:0/1648243423 <== mon.1 v2:172.21.15.176:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7ff054027d00 con 0x7ff050003bd0 2024-09-10T20:26:48.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff060ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:26:48.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff060ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:48.007 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":12,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 10 librados: watch_flush enter 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 10 librados: watch_flush exit 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 20 client.4285.objecter shutdown clearing up homeless session... 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 10 client.4285.objecter successfully canceled tick 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff04403d7a0 msgr2=0x7ff04403fc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 --2- 172.21.15.153:0/1648243423 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff04403d7a0 0x7ff04403fc60 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff04c003cc0 tx=0x7ff04c003bf0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 10 monclient: shutdown 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff050003bd0 msgr2=0x7ff06412f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 --2- 172.21.15.153:0/1648243423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff050003bd0 0x7ff06412f520 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff05400e620 tx=0x7ff0540146c0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 shutdown_connections 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 --2- 172.21.15.153:0/1648243423 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff0500099c0 0x7ff0641bffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 --2- 172.21.15.153:0/1648243423 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff04403d7a0 0x7ff04403fc60 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 --2- 172.21.15.153:0/1648243423 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff050004810 0x7ff06412fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 --2- 172.21.15.153:0/1648243423 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff050003bd0 0x7ff06412f520 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 >> 172.21.15.153:0/1648243423 conn(0x7ff0640a71b0 msgr2=0x7ff0640a4cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 shutdown_connections 2024-09-10T20:26:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 -- 172.21.15.153:0/1648243423 wait complete. 2024-09-10T20:26:48.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.005+0000 7ff069f27640 1 librados: shutdown 2024-09-10T20:26:48.024 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T20:26:48.025 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T20:26:48.025 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:26:48.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient: get_monmap_and_config 2024-09-10T20:26:48.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient: build_initial_monmap 2024-09-10T20:26:48.134 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient: monmap: 2024-09-10T20:26:48.134 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:48.134 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.134 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:48.133891+0000 2024-09-10T20:26:48.135 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:48.133891+0000 2024-09-10T20:26:48.135 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:48.135 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.135 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:48.135 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:48.135 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:48.135 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:48.135 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:48.135 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.137 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 1 Processor -- start 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 1 -- start start 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient: init 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e601364f0 0x7f7e601368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient(hunting): picked mon.noname-b con 0x7f7e601364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7e601360a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7e600a1990 0x7f7e6009d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient(hunting): picked mon.noname-c con 0x7f7e600a1990 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e600a27a0 con 0x7f7e601360a0 2024-09-10T20:26:48.138 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7e600a2a80 con 0x7f7e600a1990 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7e600a2d60 con 0x7f7e601364f0 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e66d2f640 10 monclient(hunting): authenticate will time out at 968.273493s 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e64acb640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e601364f0 msgr2=0x7f7e601368d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e64acb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e601364f0 0x7f7e601368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e57fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f7e601360a0 0x7f7e6009ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46542/0 (socket says 172.21.15.153:46542) 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e57fff640 1 -- 172.21.15.153:0/3431210926 learned_addr learned my addr 172.21.15.153:0/3431210926 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 1 --2- 172.21.15.153:0/3431210926 >> v2:172.21.15.176:3300/0 conn(0x7f7e600a1990 0x7f7e6009d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient(hunting): get_auth_request con 0x7f7e600a1990 auth_method 0 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.139 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 1 -- 172.21.15.153:0/3431210926 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e601364f0 msgr2=0x7f7e601368d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 1 --2- 172.21.15.153:0/3431210926 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e601364f0 0x7f7e601368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 1 -- 172.21.15.153:0/3431210926 >> v1:172.21.15.153:6789/0 conn(0x7f7e601360a0 legacy=0x7f7e6009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient: _finish_hunting 0 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 1 monclient: found mon.noname-c 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:48.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:26:48.141 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.133+0000 7f7e652cc640 1 -- 172.21.15.153:0/3431210926 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e60136e10 con 0x7f7e600a1990 2024-09-10T20:26:48.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e652cc640 1 -- 172.21.15.153:0/3431210926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e600a1990 msgr2=0x7f7e6009d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:48.142 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e652cc640 1 --2- 172.21.15.153:0/3431210926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e600a1990 0x7f7e6009d520 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009d90 tx=0x7f7e5801c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dffd63676921f44 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 1 -- 172.21.15.153:0/3431210926 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e580b2070 con 0x7f7e600a1990 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 10 monclient: dump: 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.143 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 10 monclient: _finish_auth 0 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 10 monclient: _check_auth_tickets 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 1 -- 172.21.15.153:0/3431210926 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6009dc70 con 0x7f7e600a1990 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 1 -- 172.21.15.153:0/3431210926 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e580a5bb0 con 0x7f7e600a1990 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 5 monclient: authenticate success, global_id 4288 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 4 set_mon_vals no callback set 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:48.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e577fe640 10 monclient: dump: 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 10 monclient: shutdown 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:48.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 1 -- 172.21.15.153:0/3431210926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e600a1990 msgr2=0x7f7e6009d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 1 --2- 172.21.15.153:0/3431210926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e600a1990 0x7f7e6009d520 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7e58009d90 tx=0x7f7e5801c850 comp rx=0 tx=0).stop 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 1 -- 172.21.15.153:0/3431210926 shutdown_connections 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 1 --2- 172.21.15.153:0/3431210926 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e600a1990 0x7f7e6009d520 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 1 --2- 172.21.15.153:0/3431210926 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e601364f0 0x7f7e601368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 1 -- 172.21.15.153:0/3431210926 >> 172.21.15.153:0/3431210926 conn(0x7f7e600a71b0 msgr2=0x7f7e600a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 1 -- 172.21.15.153:0/3431210926 shutdown_connections 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.137+0000 7f7e66d2f640 1 -- 172.21.15.153:0/3431210926 wait complete. 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient: build_initial_monmap 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient: monmap: 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.146 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 librados: starting msgr at 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 librados: starting objecter 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 Processor -- start 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 -- start start 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 librados: setting wanted keys 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 librados: calling monclient init 2024-09-10T20:26:48.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient: init 2024-09-10T20:26:48.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:48.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:26:48.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e600a1990 0x7f7e600730f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient(hunting): picked mon.noname-c con 0x7f7e600a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:48.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601360a0 0x7f7e60073630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient(hunting): picked mon.noname-a con 0x7f7e601360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:48.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e601364f0 0x7f7e60073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient(hunting): picked mon.noname-b con 0x7f7e601364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6009e6c0 con 0x7f7e601360a0 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6009e540 con 0x7f7e601364f0 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7e6009e840 con 0x7f7e600a1990 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e66d2f640 10 monclient(hunting): authenticate will time out at 968.281207s 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e64acb640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e600a1990 msgr2=0x7f7e600730f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e64acb640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e600a1990 0x7f7e600730f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601360a0 0x7f7e60073630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601360a0 0x7f7e60073630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40762/0 (socket says 172.21.15.153:40762) 2024-09-10T20:26:48.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 -- 172.21.15.153:0/1986998942 learned_addr learned my addr 172.21.15.153:0/1986998942 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient(hunting): get_auth_request con 0x7f7e601360a0 auth_method 0 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e652cc640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e601364f0 0x7f7e60073b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e652cc640 10 monclient(hunting): get_auth_request con 0x7f7e601364f0 auth_method 0 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e652cc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e652cc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e652cc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e652cc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.155 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e652cc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e652cc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e600a1990 msgr2=0x7f7e600730f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7e600a1990 0x7f7e600730f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e601364f0 msgr2=0x7f7e60073b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7e601364f0 0x7f7e60073b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient: _finish_hunting 0 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 monclient: found mon.noname-a 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 -- 172.21.15.153:0/1986998942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e60139a50 con 0x7f7e601360a0 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601360a0 msgr2=0x7f7e60073630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e57fff640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601360a0 0x7f7e60073630 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000c470 tx=0x7f7e5000c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e557fa640 10 client.?.objecter ms_handle_connect 0x7f7e601360a0 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.141+0000 7f7e557fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e50007070 con 0x7f7e601360a0 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: dump: 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.157 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: _finish_auth 0 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: _check_auth_tickets 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 5 monclient: authenticate success, global_id 4344 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 10 monclient: _renew_subs 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e601372f0 con 0x7f7e601360a0 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e50007210 con 0x7f7e601360a0 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e50007600 con 0x7f7e601360a0 2024-09-10T20:26:48.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: dump: 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:48.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 10 monclient: _renew_subs 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e6009e0a0 con 0x7f7e601360a0 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 1 librados: init done 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e56ffd640 4 set_mon_vals no callback set 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601392b0 0x7f7e601cc9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e6009dc70 con 0x7f7e601392b0 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601392b0 0x7f7e601cc9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 10 monclient: get_auth_request con 0x7f7e601392b0 auth_method 0 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 10 monclient(hunting): handle_auth_done global_id 4347 payload 824 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f7e5001b400 con 0x7f7e601360a0 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 1 -- 172.21.15.153:0/1986998942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7e601368d0 con 0x7f7e601392b0 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 --2- 172.21.15.153:0/1986998942 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7e4c03ce50 0x7f7e4c03f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f7e5004efa0 con 0x7f7e601360a0 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 client.4344.objecter ms_dispatch 0x7f7e60000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 3 client.4344.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 3 client.4344.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 20 client.4344.objecter dump_active .. 0 homeless 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e652cc640 1 --2- 172.21.15.153:0/1986998942 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7e4c03ce50 0x7f7e4c03f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e652cc640 10 monclient: get_auth_request con 0x7f7e4c03ce50 auth_method 0 2024-09-10T20:26:48.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601392b0 msgr2=0x7f7e601cc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 client.4344.objecter ms_handle_connect 0x7f7e601392b0 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e64acb640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601392b0 0x7f7e601cc9e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00b450 tx=0x7f7e5c00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 client.4344.objecter resend_mon_ops 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c005270 con 0x7f7e601392b0 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: dump: 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.162 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7e5c015070 con 0x7f7e601392b0 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601392b0 msgr2=0x7f7e601cc9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601392b0 0x7f7e601cc9e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00b450 tx=0x7f7e5c00b910 comp rx=0 tx=0).stop 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601392b0 msgr2=0x7f7e601cc9e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.145+0000 7f7e557fa640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601392b0 0x7f7e601cc9e0 secure :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00b450 tx=0x7f7e5c00b910 comp rx=0 tx=0).stop 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.149+0000 7f7e557fa640 10 client.4344.objecter ms_handle_connect 0x7f7e4c03ce50 2024-09-10T20:26:48.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.149+0000 7f7e652cc640 1 --2- 172.21.15.153:0/1986998942 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7e4c03ce50 0x7f7e4c03f310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e580af000 tx=0x7f7e580a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e66d2f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:26:48.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e66d2f640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601ccf20 0x7f7e601cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e66d2f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7e601cd870 con 0x7f7e601ccf20 2024-09-10T20:26:48.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601ccf20 0x7f7e601cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 10 monclient: get_auth_request con 0x7f7e601ccf20 auth_method 0 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 10 monclient(hunting): handle_auth_done global_id 4350 payload 824 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.153+0000 7f7e64acb640 1 -- 172.21.15.153:0/1986998942 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7e6006a660 con 0x7f7e601ccf20 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e64acb640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601ccf20 msgr2=0x7f7e601cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 10 client.4344.objecter ms_handle_connect 0x7f7e601ccf20 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 10 client.4344.objecter resend_mon_ops 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e64acb640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601ccf20 0x7f7e601cd330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00b450 tx=0x7f7e5c0059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.166 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c01ae60 con 0x7f7e601ccf20 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 10 monclient: dump: 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7f7e5c010480 con 0x7f7e601ccf20 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601ccf20 msgr2=0x7f7e601cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601ccf20 0x7f7e601cd330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00b450 tx=0x7f7e5c0059e0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601ccf20 msgr2=0x7f7e601cd330 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e557fa640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601ccf20 0x7f7e601cd330 secure :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00b450 tx=0x7f7e5c0059e0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:26:48.168 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 12, 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 54847, 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:48.169 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:48.170 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:26:48.171 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:26:48.172 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.173 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:26:48.174 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:48.175 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:26:48.176 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.177 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.178 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:26:48.179 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:26:48.180 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:26:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 10 librados: watch_flush enter 2024-09-10T20:26:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 10 librados: watch_flush exit 2024-09-10T20:26:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 20 client.4344.objecter shutdown clearing up homeless session... 2024-09-10T20:26:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 10 client.4344.objecter successfully canceled tick 2024-09-10T20:26:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7e4c03ce50 msgr2=0x7f7e4c03f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 1 --2- 172.21.15.153:0/1986998942 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7e4c03ce50 0x7f7e4c03f310 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f7e580af000 tx=0x7f7e580a7070 comp rx=0 tx=0).stop 2024-09-10T20:26:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 10 monclient: shutdown 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601360a0 msgr2=0x7f7e60073630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 1 --2- 172.21.15.153:0/1986998942 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7e601360a0 0x7f7e60073630 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f7e5000c470 tx=0x7f7e5000c930 comp rx=0 tx=0).stop 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e652cc640 1 -- 172.21.15.153:0/1986998942 reap_dead start 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e652cc640 1 -- 172.21.15.153:0/1986998942 reap_dead start 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 shutdown_connections 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 >> 172.21.15.153:0/1986998942 conn(0x7f7e600a71b0 msgr2=0x7f7e600a6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 shutdown_connections 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 1 -- 172.21.15.153:0/1986998942 wait complete. 2024-09-10T20:26:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.157+0000 7f7e66d2f640 1 librados: shutdown 2024-09-10T20:26:48.183 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T20:26:48.184 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T20:26:48.184 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient: get_monmap_and_config 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient: build_initial_monmap 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient: monmap: 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:26:48.294492+0000 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:26:48.294492+0000 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:26:48.295 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:26:48.296 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:26:48.296 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:26:48.296 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 1 Processor -- start 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 1 -- start start 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient: init 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient: _add_conns ranks=[4,1,0,2,3] 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f39f80a41a0 0x7f39f80a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient(hunting): picked mon.noname-c con 0x7f39f80a41a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39f80a4580 addr v1:172.21.15.176:6789/0 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f39f80a49d0 addr v1:172.21.15.153:6789/0 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f80a67f0 con 0x7f39f80a49d0 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39f80a6a20 con 0x7f39f80a4580 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f39f80a2570 con 0x7f39f80a41a0 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f3a00476640 10 monclient(hunting): authenticate will time out at 968.434759s 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fea13640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f39f80a49d0 0x7f39f80a60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:46556/0 (socket says 172.21.15.153:46556) 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fea13640 1 -- 172.21.15.153:0/1914666238 learned_addr learned my addr 172.21.15.153:0/1914666238 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 1 --2- 172.21.15.153:0/1914666238 >> v2:172.21.15.176:3300/0 conn(0x7f39f80a41a0 0x7f39f80a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient(hunting): get_auth_request con 0x7f39f80a41a0 auth_method 0 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient(hunting): handle_auth_done global_id 4294 payload 274 2024-09-10T20:26:48.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 1 -- 172.21.15.153:0/1914666238 >> v1:172.21.15.176:6789/0 conn(0x7f39f80a4580 legacy=0x7f39f80a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:48.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 1 -- 172.21.15.153:0/1914666238 >> v1:172.21.15.153:6789/0 conn(0x7f39f80a49d0 legacy=0x7f39f80a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:26:48.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient: _finish_hunting 0 2024-09-10T20:26:48.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 1 monclient: found mon.noname-c 2024-09-10T20:26:48.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:48.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:26:48.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.293+0000 7f39fe212640 1 -- 172.21.15.153:0/1914666238 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f80a3c80 con 0x7f39f80a41a0 2024-09-10T20:26:48.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 -- 172.21.15.153:0/1914666238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80a41a0 msgr2=0x7f39f80a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 --2- 172.21.15.153:0/1914666238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80a41a0 0x7f39f80a3740 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f39e80097c0 tx=0x7f39e801c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83cd9eced1b5e5ee server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 1 -- 172.21.15.153:0/1914666238 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e80b0070 con 0x7f39f80a41a0 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 10 monclient: dump: 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.303 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 10 monclient: _finish_auth 0 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 10 monclient: _check_auth_tickets 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 1 -- 172.21.15.153:0/1914666238 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e80ab600 con 0x7f39f80a41a0 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 1 -- 172.21.15.153:0/1914666238 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e80ab9d0 con 0x7f39f80a41a0 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 5 monclient: authenticate success, global_id 4294 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient: get_monmap_and_config success 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 4 set_mon_vals no callback set 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fd210640 10 monclient: dump: 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.305 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient: shutdown 2024-09-10T20:26:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 -- 172.21.15.153:0/1914666238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80a41a0 msgr2=0x7f39f80a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 --2- 172.21.15.153:0/1914666238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80a41a0 0x7f39f80a3740 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f39e80097c0 tx=0x7f39e801c360 comp rx=0 tx=0).stop 2024-09-10T20:26:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 -- 172.21.15.153:0/1914666238 shutdown_connections 2024-09-10T20:26:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 --2- 172.21.15.153:0/1914666238 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80a41a0 0x7f39f80a3740 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 -- 172.21.15.153:0/1914666238 >> 172.21.15.153:0/1914666238 conn(0x7f39f80ab7e0 msgr2=0x7f39f80a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:48.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 -- 172.21.15.153:0/1914666238 shutdown_connections 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 -- 172.21.15.153:0/1914666238 wait complete. 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient: build_initial_monmap 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient: monmap: 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 librados: starting msgr at 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 librados: starting objecter 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 Processor -- start 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 -- start start 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 librados: setting wanted keys 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 librados: calling monclient init 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient: init 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient: _reopen_session rank -1 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39f80a41a0 0x7f39f8128a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient(hunting): picked mon.noname-a con 0x7f39f80a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80a4580 0x7f39f8140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient(hunting): picked mon.noname-b con 0x7f39f80a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39f80a49d0 0x7f39f81d0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient(hunting): picked mon.noname-c con 0x7f39f80a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39f8144630 con 0x7f39f80a41a0 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f39f8070400 con 0x7f39f80a4580 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f39f81447b0 con 0x7f39f80a49d0 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient(hunting): _renew_subs 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f3a00476640 10 monclient(hunting): authenticate will time out at 968.443240s 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fea13640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39f80a49d0 msgr2=0x7f39f81d0990 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fea13640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39f80a49d0 0x7f39f81d0990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39f80a41a0 0x7f39f8128a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39f80a41a0 0x7f39f8128a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:40794/0 (socket says 172.21.15.153:40794) 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 -- 172.21.15.153:0/2466882171 learned_addr learned my addr 172.21.15.153:0/2466882171 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient(hunting): get_auth_request con 0x7f39f80a41a0 auth_method 0 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fda11640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80a4580 0x7f39f8140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fda11640 10 monclient(hunting): get_auth_request con 0x7f39f80a4580 auth_method 0 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fda11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fda11640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fda11640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fda11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fda11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fda11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient(hunting): handle_auth_done global_id 4353 payload 824 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39f80a49d0 msgr2=0x7f39f81d0990 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f39f80a49d0 0x7f39f81d0990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80a4580 msgr2=0x7f39f8140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f80a4580 0x7f39f8140880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient: _finish_hunting 0 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 monclient: found mon.noname-a 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:26:48.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:26:48.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.301+0000 7f39fe212640 1 -- 172.21.15.153:0/2466882171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39f8129f20 con 0x7f39f80a41a0 2024-09-10T20:26:48.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.305+0000 7f39fe212640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39f80a41a0 msgr2=0x7f39f8128a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:48.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.305+0000 7f39fe212640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39f80a41a0 0x7f39f8128a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f39e801aa10 tx=0x7f39e8019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.305+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 DispatchQueue::entry inject delay of 0.002000 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 client.?.objecter ms_handle_connect 0x7f39f80a41a0 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e80b9020 con 0x7f39f80a41a0 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 monclient: dump: 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.318 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 monclient: _finish_auth 0 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 monclient: _check_auth_tickets 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39e80b5aa0 con 0x7f39f80a41a0 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 5 monclient: authenticate success, global_id 4353 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 10 monclient: _renew_subs 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39f80a21c0 con 0x7f39f80a41a0 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:26:48.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39e80b0040 con 0x7f39f80a41a0 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 monclient: dump: 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fca0f640 4 set_mon_vals no callback set 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 10 monclient: _renew_subs 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39f8127ea0 con 0x7f39f80a41a0 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 1 librados: init done 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81268a0 0x7f39f81d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f39f812a160 con 0x7f39f81268a0 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f39e80bc020 con 0x7f39f80a41a0 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81268a0 0x7f39f81d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 10 monclient: get_auth_request con 0x7f39f81268a0 auth_method 0 2024-09-10T20:26:48.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 1 --2- 172.21.15.153:0/2466882171 >> v2:172.21.15.176:6812/3003473398 conn(0x7f39dc03ced0 0x7f39dc03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3562+0+0 (secure 0 0 0) 0x7f39e80e9690 con 0x7f39f80a41a0 2024-09-10T20:26:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 client.4353.objecter ms_dispatch 0x7f39f8000ba0 osd_map(20..20 src has 1..20) 2024-09-10T20:26:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 3 client.4353.objecter handle_osd_map got epochs [20,20] > 0 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 3 client.4353.objecter handle_osd_map decoding full epoch 20 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 20 client.4353.objecter dump_active .. 0 homeless 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 10 monclient(hunting): handle_auth_done global_id 4300 payload 824 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fea13640 1 --2- 172.21.15.153:0/2466882171 >> v2:172.21.15.176:6812/3003473398 conn(0x7f39dc03ced0 0x7f39dc03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 1 -- 172.21.15.153:0/2466882171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39f80a3720 con 0x7f39f81268a0 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fea13640 10 monclient: get_auth_request con 0x7f39dc03ced0 auth_method 0 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81268a0 msgr2=0x7f39f81d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 client.4353.objecter ms_handle_connect 0x7f39f81268a0 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39eeffd640 10 client.4353.objecter resend_mon_ops 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.309+0000 7f39fda11640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81268a0 0x7f39f81d0ed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39f80a11c0 tx=0x7f39f400fc00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f4003b80 con 0x7f39f81268a0 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 10 monclient: dump: 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 10 client.4353.objecter ms_handle_connect 0x7f39dc03ced0 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39fea13640 1 --2- 172.21.15.153:0/2466882171 >> v2:172.21.15.176:6812/3003473398 conn(0x7f39dc03ced0 0x7f39dc03f390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f39f8128790 tx=0x7f39f0004fe0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f39f4019070 con 0x7f39f81268a0 2024-09-10T20:26:48.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:26:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:26:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81268a0 msgr2=0x7f39f81d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81268a0 0x7f39f81d0ed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39f80a11c0 tx=0x7f39f400fc00 comp rx=0 tx=0).stop 2024-09-10T20:26:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81268a0 msgr2=0x7f39f81d0ed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.313+0000 7f39eeffd640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81268a0 0x7f39f81d0ed0 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f39f80a11c0 tx=0x7f39f400fc00 comp rx=0 tx=0).stop 2024-09-10T20:26:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f3a00476640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:26:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f3a00476640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81d1410 0x7f39f8121620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f3a00476640 10 monclient(hunting): start opening mon connection 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f39f81d1820 con 0x7f39f81d1410 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81d1410 0x7f39f8121620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 10 monclient: get_auth_request con 0x7f39f81d1410 auth_method 0 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 10 monclient(hunting): handle_auth_done global_id 4303 payload 824 2024-09-10T20:26:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.317+0000 7f39fda11640 1 -- 172.21.15.153:0/2466882171 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f39f8127820 con 0x7f39f81d1410 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39fda11640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81d1410 msgr2=0x7f39f8121620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 10 client.4353.objecter ms_handle_connect 0x7f39f81d1410 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 10 client.4353.objecter resend_mon_ops 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39fda11640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81d1410 0x7f39f8121620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f39f80a11c0 tx=0x7f39f4003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39f4006e00 con 0x7f39f81d1410 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 10 monclient: dump: 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:26:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4875 (secure 0 0 0) 0x7f39f4006e00 con 0x7f39f81d1410 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81d1410 msgr2=0x7f39f8121620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81d1410 0x7f39f8121620 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f39f80a11c0 tx=0x7f39f4003770 comp rx=0 tx=0).stop 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81d1410 msgr2=0x7f39f8121620 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:26:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39eeffd640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39f81d1410 0x7f39f8121620 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f39f80a11c0 tx=0x7f39f4003770 comp rx=0 tx=0).stop 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 12, 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 54963, 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.331 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:48.332 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:48.333 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:26:48.334 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:26:48.335 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:48.336 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.337 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:26:48.338 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:48.339 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.340 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.341 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.342 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:26:48.343 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:26:48.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:26:48.344 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:26:48.344 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:26:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 10 librados: watch_flush enter 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 10 librados: watch_flush exit 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 20 client.4353.objecter shutdown clearing up homeless session... 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 10 client.4353.objecter successfully canceled tick 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 >> v2:172.21.15.176:6812/3003473398 conn(0x7f39dc03ced0 msgr2=0x7f39dc03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 --2- 172.21.15.153:0/2466882171 >> v2:172.21.15.176:6812/3003473398 conn(0x7f39dc03ced0 0x7f39dc03f390 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f39f8128790 tx=0x7f39f0004fe0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 10 monclient: shutdown 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39f80a41a0 msgr2=0x7f39f8128a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f39fea13640 1 -- 172.21.15.153:0/2466882171 reap_dead start 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39f80a41a0 0x7f39f8128a00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f39e801aa10 tx=0x7f39e8019100 comp rx=0 tx=0).stop 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 shutdown_connections 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 --2- 172.21.15.153:0/2466882171 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39f80a41a0 0x7f39f8128a00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:26:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 >> 172.21.15.153:0/2466882171 conn(0x7f39f80ab7e0 msgr2=0x7f39f80a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:26:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 shutdown_connections 2024-09-10T20:26:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 -- 172.21.15.153:0/2466882171 wait complete. 2024-09-10T20:26:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:26:48.321+0000 7f3a00476640 1 librados: shutdown 2024-09-10T20:26:48.347 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T20:26:48.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-10T20:27:14.860 INFO:tasks.workunit.client.0.smithi176.stderr:Note: switching to 'bad1f558c327094897e4a32e99dd5cb4ae6ccd46'. 2024-09-10T20:27:14.860 INFO:tasks.workunit.client.0.smithi176.stderr: 2024-09-10T20:27:14.860 INFO:tasks.workunit.client.0.smithi176.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-10T20:27:14.860 INFO:tasks.workunit.client.0.smithi176.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-10T20:27:14.860 INFO:tasks.workunit.client.0.smithi176.stderr:state without impacting any branches by switching back to a branch. 2024-09-10T20:27:14.860 INFO:tasks.workunit.client.0.smithi176.stderr: 2024-09-10T20:27:14.860 INFO:tasks.workunit.client.0.smithi176.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-10T20:27:14.860 INFO:tasks.workunit.client.0.smithi176.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-10T20:27:14.860 INFO:tasks.workunit.client.0.smithi176.stderr: 2024-09-10T20:27:14.861 INFO:tasks.workunit.client.0.smithi176.stderr: git switch -c 2024-09-10T20:27:14.861 INFO:tasks.workunit.client.0.smithi176.stderr: 2024-09-10T20:27:14.861 INFO:tasks.workunit.client.0.smithi176.stderr:Or undo this operation with: 2024-09-10T20:27:14.861 INFO:tasks.workunit.client.0.smithi176.stderr: 2024-09-10T20:27:14.861 INFO:tasks.workunit.client.0.smithi176.stderr: git switch - 2024-09-10T20:27:14.861 INFO:tasks.workunit.client.0.smithi176.stderr: 2024-09-10T20:27:14.861 INFO:tasks.workunit.client.0.smithi176.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-10T20:27:14.861 INFO:tasks.workunit.client.0.smithi176.stderr: 2024-09-10T20:27:14.861 INFO:tasks.workunit.client.0.smithi176.stderr:HEAD is now at bad1f558c32 Merge branch 'wip-ifed-bluestore-tool-really-really' of https://github.com/ifed01/ceph into wip-bharath7-testing-2024-09-10-1409 2024-09-10T20:27:14.866 DEBUG:teuthology.orchestra.run.smithi176:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-10T20:27:14.914 INFO:tasks.workunit.client.0.smithi176.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-10T20:27:14.917 INFO:tasks.workunit.client.0.smithi176.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T20:27:14.918 INFO:tasks.workunit.client.0.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-10T20:27:15.742 INFO:tasks.workunit.client.0.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-10T20:27:15.828 INFO:tasks.workunit.client.0.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-10T20:27:15.880 INFO:tasks.workunit.client.0.smithi176.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-10T20:27:15.882 INFO:tasks.workunit.client.0.smithi176.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T20:27:15.882 INFO:tasks.workunit.client.0.smithi176.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-10T20:27:15.927 INFO:tasks.workunit.client.0.smithi176.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-10T20:27:15.931 DEBUG:teuthology.orchestra.run.smithi176:> set -ex 2024-09-10T20:27:15.931 DEBUG:teuthology.orchestra.run.smithi176:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-10T20:27:15.978 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-10T20:27:15.979 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-10T20:27:15.979 DEBUG:teuthology.orchestra.run.smithi176:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=bad1f558c327094897e4a32e99dd5cb4ae6ccd46 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-10T20:27:16.031 INFO:tasks.workunit.client.0.smithi176.stderr:+ parallel=1 2024-09-10T20:27:16.031 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' '' = --serial ']' 2024-09-10T20:27:16.031 INFO:tasks.workunit.client.0.smithi176.stderr:+ crimson=0 2024-09-10T20:27:16.031 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' '' = --crimson ']' 2024-09-10T20:27:16.031 INFO:tasks.workunit.client.0.smithi176.stderr:+ color= 2024-09-10T20:27:16.031 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -t 1 ']' 2024-09-10T20:27:16.031 INFO:tasks.workunit.client.0.smithi176.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-10T20:27:16.031 INFO:tasks.workunit.client.0.smithi176.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-10T20:27:16.031 INFO:tasks.workunit.client.0.smithi176.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-10T20:27:16.032 INFO:tasks.workunit.client.0.smithi176.stderr:+ declare -A pids 2024-09-10T20:27:16.032 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.032 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.032 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_aio 2024-09-10T20:27:16.032 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_aio' 2024-09-10T20:27:16.033 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_aio 2024-09-10T20:27:16.033 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.035 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_aio 2024-09-10T20:27:16.036 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27928 2024-09-10T20:27:16.036 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_aio on pid 27928' 2024-09-10T20:27:16.036 INFO:tasks.workunit.client.0.smithi176.stdout:test api_aio on pid 27928 2024-09-10T20:27:16.037 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=27928 2024-09-10T20:27:16.037 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.037 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.037 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_aio_pp 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_aio_pp' 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_aio_pp 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.038 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_aio: /' 2024-09-10T20:27:16.039 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_aio_pp 2024-09-10T20:27:16.039 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27936 2024-09-10T20:27:16.040 INFO:tasks.workunit.client.0.smithi176.stdout:test api_aio_pp on pid 27936 2024-09-10T20:27:16.041 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_aio_pp on pid 27936' 2024-09-10T20:27:16.041 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=27936 2024-09-10T20:27:16.041 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.041 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.041 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-10T20:27:16.041 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_io 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_io' 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_io 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-10T20:27:16.042 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-10T20:27:16.043 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_io 2024-09-10T20:27:16.043 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27944 2024-09-10T20:27:16.043 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_io on pid 27944' 2024-09-10T20:27:16.043 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=27944 2024-09-10T20:27:16.043 INFO:tasks.workunit.client.0.smithi176.stdout:test api_io on pid 27944 2024-09-10T20:27:16.044 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_io_pp 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_io_pp' 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_io_pp 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_io.log 2024-09-10T20:27:16.045 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_io: /' 2024-09-10T20:27:16.046 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_io_pp 2024-09-10T20:27:16.046 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27952 2024-09-10T20:27:16.046 INFO:tasks.workunit.client.0.smithi176.stdout:test api_io_pp on pid 27952 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_io_pp on pid 27952' 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=27952 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_asio 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_asio' 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.048 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_asio 2024-09-10T20:27:16.049 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-10T20:27:16.049 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.049 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-10T20:27:16.049 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-10T20:27:16.049 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_asio 2024-09-10T20:27:16.050 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27960 2024-09-10T20:27:16.050 INFO:tasks.workunit.client.0.smithi176.stdout:test api_asio on pid 27960 2024-09-10T20:27:16.050 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_asio on pid 27960' 2024-09-10T20:27:16.051 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=27960 2024-09-10T20:27:16.051 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.051 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.051 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-10T20:27:16.051 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_list 2024-09-10T20:27:16.051 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_list' 2024-09-10T20:27:16.052 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_list 2024-09-10T20:27:16.052 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.052 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.053 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.053 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.053 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.053 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-10T20:27:16.053 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-10T20:27:16.053 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_asio: /' 2024-09-10T20:27:16.053 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_list 2024-09-10T20:27:16.054 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27968 2024-09-10T20:27:16.054 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_list on pid 27968' 2024-09-10T20:27:16.054 INFO:tasks.workunit.client.0.smithi176.stdout:test api_list on pid 27968 2024-09-10T20:27:16.055 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=27968 2024-09-10T20:27:16.055 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.055 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.055 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-10T20:27:16.055 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_lock 2024-09-10T20:27:16.055 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_lock' 2024-09-10T20:27:16.056 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_lock 2024-09-10T20:27:16.056 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.056 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.056 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.056 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.056 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.056 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-10T20:27:16.056 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_list: /' 2024-09-10T20:27:16.057 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_list.log 2024-09-10T20:27:16.058 INFO:tasks.workunit.client.0.smithi176.stdout:test api_lock on pid 27976 2024-09-10T20:27:16.058 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_lock 2024-09-10T20:27:16.058 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27976 2024-09-10T20:27:16.058 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_lock on pid 27976' 2024-09-10T20:27:16.058 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=27976 2024-09-10T20:27:16.059 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.059 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.059 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-10T20:27:16.059 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.060 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.060 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.060 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_lock_pp 2024-09-10T20:27:16.060 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.060 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_lock_pp' 2024-09-10T20:27:16.062 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_lock_pp 2024-09-10T20:27:16.062 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_lock: /' 2024-09-10T20:27:16.063 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-10T20:27:16.063 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-10T20:27:16.073 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.076 INFO:tasks.workunit.client.0.smithi176.stdout:test api_lock_pp on pid 27988 2024-09-10T20:27:16.077 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_lock_pp 2024-09-10T20:27:16.077 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27988 2024-09-10T20:27:16.077 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_lock_pp on pid 27988' 2024-09-10T20:27:16.077 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=27988 2024-09-10T20:27:16.077 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.077 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.077 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-10T20:27:16.077 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_misc 2024-09-10T20:27:16.077 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_misc' 2024-09-10T20:27:16.078 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.078 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.078 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.078 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.078 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-10T20:27:16.078 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-10T20:27:16.079 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_misc 2024-09-10T20:27:16.080 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-10T20:27:16.082 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.085 INFO:tasks.workunit.client.0.smithi176.stdout:test api_misc on pid 28015 2024-09-10T20:27:16.085 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_misc 2024-09-10T20:27:16.086 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28015 2024-09-10T20:27:16.086 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_misc on pid 28015' 2024-09-10T20:27:16.086 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28015 2024-09-10T20:27:16.086 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.086 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.086 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_misc_pp 2024-09-10T20:27:16.086 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_misc_pp' 2024-09-10T20:27:16.086 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-10T20:27:16.087 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.087 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.087 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.087 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.087 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-10T20:27:16.088 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-10T20:27:16.088 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_misc: /' 2024-09-10T20:27:16.088 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_misc_pp 2024-09-10T20:27:16.088 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.092 INFO:tasks.workunit.client.0.smithi176.stdout:test api_misc_pp on pid 28039 2024-09-10T20:27:16.092 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_misc_pp 2024-09-10T20:27:16.093 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28039 2024-09-10T20:27:16.093 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_misc_pp on pid 28039' 2024-09-10T20:27:16.093 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28039 2024-09-10T20:27:16.093 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.093 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.095 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_pool 2024-09-10T20:27:16.095 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_pool' 2024-09-10T20:27:16.095 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-10T20:27:16.096 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_pool 2024-09-10T20:27:16.096 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.097 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.097 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.097 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.097 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.098 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-10T20:27:16.098 INFO:tasks.workunit.client.0.smithi176.stdout:test api_pool on pid 28063 2024-09-10T20:27:16.098 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_pool 2024-09-10T20:27:16.099 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28063 2024-09-10T20:27:16.099 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_pool on pid 28063' 2024-09-10T20:27:16.099 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28063 2024-09-10T20:27:16.099 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.099 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.105 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-10T20:27:16.105 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [----------] Global test environment set-up. 2024-09-10T20:27:16.105 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient: get_monmap_and_config 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient: build_initial_monmap 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient: monmap: 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: epoch 0 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: last_changed 0.000000 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: created 0.000000 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: election_strategy: 1 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:27:16.106 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 1 Processor -- start 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 1 -- start start 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient: init 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient: _reopen_session rank -1 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 0x56266d283c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient(hunting): picked mon.noname-c con 0x56266d2a5450 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56266d2a5830 0x56266d282f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient(hunting): picked mon.noname-a con 0x56266d2a5830 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d333ef0 0x56266d281a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient(hunting): picked mon.noname-b con 0x56266d333ef0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient(hunting): start opening mon connection 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56266d1d79f0 con 0x56266d2a5830 2024-09-10T20:27:16.107 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient(hunting): start opening mon connection 2024-09-10T20:27:16.108 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x56266d235f30 con 0x56266d333ef0 2024-09-10T20:27:16.108 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient(hunting): start opening mon connection 2024-09-10T20:27:16.108 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x56266d337360 con 0x56266d2a5450 2024-09-10T20:27:16.108 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient(hunting): _renew_subs 2024-09-10T20:27:16.108 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.095+0000 7f7b7d13c880 10 monclient(hunting): authenticate will time out at 1000.971952s 2024-09-10T20:27:16.108 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_snapshots 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7ac46640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 msgr2=0x56266d283c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7ac46640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 0x56266d283c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d333ef0 0x56266d281a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d333ef0 0x56266d281a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:56492/0 (socket says 172.21.15.176:56492) 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 1 -- 172.21.15.176:0/376869779 learned_addr learned my addr 172.21.15.176:0/376869779 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient(hunting): get_auth_request con 0x56266d333ef0 auth_method 0 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:27:16.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:27:16.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2024-09-10T20:27:16.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 1 -- 172.21.15.176:0/376869779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 msgr2=0x56266d283c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:27:16.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 1 --2- 172.21.15.176:0/376869779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 0x56266d283c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:27:16.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 1 -- 172.21.15.176:0/376869779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56266d2a5830 msgr2=0x56266d282f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:27:16.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 1 --2- 172.21.15.176:0/376869779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56266d2a5830 0x56266d282f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:27:16.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient: _finish_hunting 0 2024-09-10T20:27:16.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 1 monclient: found mon.noname-b 2024-09-10T20:27:16.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:27:16.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:27:16.111 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_snapshots' 2024-09-10T20:27:16.111 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-10T20:27:16.111 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_snapshots 2024-09-10T20:27:16.112 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-10T20:27:16.112 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-10T20:27:16.112 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.112 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.112 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.112 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.112 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-10T20:27:16.112 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-10T20:27:16.112 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.114 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.099+0000 7f7b7b447640 1 -- 172.21.15.176:0/376869779 --> [v2:172.21.15.176:3300/0,v1:172.21.15.1test api_snapshots on pid 28102 2024-09-10T20:27:16.115 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_snapshots 2024-09-10T20:27:16.115 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28102 2024-09-10T20:27:16.115 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_snapshots on pid 28102' 2024-09-10T20:27:16.115 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28102 2024-09-10T20:27:16.115 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.115 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.115 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_snapshots_pp 2024-09-10T20:27:16.115 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-10T20:27:16.115 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_snapshots_pp' 2024-09-10T20:27:16.116 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.116 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.116 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_snapshots_pp 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_pool: /' 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stdout:76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56266d1b0c00 con 0x56266d333ef0 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.103+0000 7f7b7b447640 1 -- 172.21.15.176:0/376869779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d333ef0 msgr2=0x56266d281a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.103+0000 7f7b7b447640 1 --2- 172.21.15.176:0/376869779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d333ef0 0x56266d281a40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b70006f90 tx=0x7f7b7000b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df07017f8dce6443 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 1 -- 172.21.15.176:0/376869779 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b70012020 con 0x56266d333ef0 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:27:16.117 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 10 monclient: dump: 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: epoch 1 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: min_mon_release 19 (squid) 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: election_strategy: 1 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 10 monclient: _finish_auth 0 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 10 monclient: _check_auth_tickets 2024-09-10T20:27:16.118 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 1 -- 172.21.15.176:0/376869779 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b7000eaa0 con 0x56266d333ef0 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 10 monclient: handle_config config(0 keys) 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 1 -- 172.21.15.176:0/376869779 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7000d3e0 con 0x56266d333ef0 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b79443640 10 monclient: dump: 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: epoch 1 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: min_mon_release 19 (squid) 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: election_strategy: 1 2024-09-10T20:27:16.119 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:27:16.120 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:27:16.120 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:27:16.120 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.120 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b7d13c880 5 monclient: authenticate success, global_id 4321 2024-09-10T20:27:16.120 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b7d13c880 10 monclient: get_monmap_and_config success 2024-09-10T20:27:16.120 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-10T20:27:16.120 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.121 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b7d13c880 4 set_mon_vals no callback set 2024-09-10T20:27:16.121 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b7d13c880 10 monclient: shutdown 2024-09-10T20:27:16.121 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b7d13c880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:27:16.121 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b7d13c880 1 -- 172.21.15.176:0/376869779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d333ef0 msgr2=0x56266d281a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:27:16.121 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.107+0000 7f7b7d13c880 1 --2- 172.21.15.176:0/376869779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d333ef0 0x56266d281a40 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7b70006f90 tx=0x7f7b7000b790 comp rx=0 tx=0).stop 2024-09-10T20:27:16.121 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 -- 172.21.15.176:0/376869779 shutdown_connections 2024-09-10T20:27:16.121 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 --2- 172.21.15.176:0/376869779 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d333ef0 0x56266d281a40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:27:16.121 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 --2- 172.21.15.176:0/376869779 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56266d2a5830 0x56266d282f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 --2- 172.21.15.176:0/376869779 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 0x56266d283c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 -- 172.21.15.176:0/376869779 >> 172.21.15.176:0/376869779 conn(0x56266d28a370 msgr2=0x56266d27ec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 -- 172.21.15.176:0/376869779 shutdown_connections 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 -- 172.21.15.176:0/376869779 wait complete. 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient: build_initial_monmap 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient: monmap: 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: epoch 0 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: last_changed 0.000000 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: created 0.000000 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: election_strategy: 1 2024-09-10T20:27:16.122 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 librados: starting msgr at 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 librados: starting objecter 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 Processor -- start 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 -- start start 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 librados: setting wanted keys 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 librados: calling monclient init 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient: init 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient: _reopen_session rank -1 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:27:16.123 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 0x56266d33a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient(hunting): picked mon.noname-c con 0x56266d2a5450 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d2a5830 0x56266d33ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient(hunting): picked mon.noname-b con 0x56266d2a5830 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56266d333ef0 0x56266d33b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient(hunting): picked mon.noname-a con 0x56266d333ef0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient(hunting): start opening mon connection 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x56266d2a6aa0 con 0x56266d333ef0 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient(hunting): start opening mon connection 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x56266d2a6460 con 0x56266d2a5830 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient(hunting): start opening mon connection 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x56266d29b310 con 0x56266d2a5450 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient(hunting): _renew_subs 2024-09-10T20:27:16.124 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7d13c880 10 monclient(hunting): authenticate will time out at 1000.986056s 2024-09-10T20:27:16.125 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7ac46640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 msgr2=0x56266d33a7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:27:16.125 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7ac46640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 0x56266d33a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:27:16.125 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d2a5830 0x56266d33ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:27:16.125 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d2a5830 0x56266d33ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.176:56528/0 (socket says 172.21.15.176:56528) 2024-09-10T20:27:16.125 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 1 -- 172.21.15.176:0/4265800331 learned_addr learned my addr 172.21.15.176:0/4265800331 (peer_addr_for_me v2:172.21.15.176:0/0) 2024-09-10T20:27:16.125 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 10 monclient(hunting): get_auth_request con 0x56266d2a5830 auth_method 0 2024-09-10T20:27:16.125 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:27:16.125 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:27:16.125 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 test api_snapshots_pp on pid 28124 2024-09-10T20:27:16.126 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_snapshots_pp 2024-09-10T20:27:16.126 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-10T20:27:16.126 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28124 2024-09-10T20:27:16.126 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_snapshots_pp on pid 28124' 2024-09-10T20:27:16.126 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28124 2024-09-10T20:27:16.126 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.127 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.127 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_stat 2024-09-10T20:27:16.127 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-10T20:27:16.127 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_stat' 2024-09-10T20:27:16.127 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-10T20:27:16.127 INFO:tasks.workunit.client.0.smithi176.stdout:7f7b7a445640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:27:16.127 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:27:16.127 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:27:16.127 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 1 -- 172.21.15.176:0/4265800331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 msgr2=0x56266d33a7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 1 --2- 172.21.15.176:0/4265800331 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x56266d2a5450 0x56266d33a7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 1 -- 172.21.15.176:0/4265800331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56266d333ef0 msgr2=0x56266d33b220 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 1 --2- 172.21.15.176:0/4265800331 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x56266d333ef0 0x56266d33b220 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 10 monclient: _finish_hunting 0 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 1 monclient: found mon.noname-b 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.176:3300/0 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.111+0000 7f7b7a445640 1 -- 172.21.15.176:0/4265800331 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56266d33d830 con 0x56266d2a5830 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7a445640 1 -- 172.21.15.176:0/4265800331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d2a5830 msgr2=0x56266d33ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7a445640 1 --2- 172.21.15.176:0/4265800331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d2a5830 0x56266d33ace0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c025c30 tx=0x7f7b6c0ae5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:27:16.128 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 client.?.objecter ms_handle_connect 0x56266d2a5830 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0bd070 con 0x56266d2a5830 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 monclient: dump: 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: epoch 1 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: min_mon_release 19 (squid) 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: election_strategy: 1 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:27:16.129 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 monclient: _finish_auth 0 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 monclient: _check_auth_tickets 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b6c0c1400 con 0x56266d2a5830 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b6c0c17f0 con 0x56266d2a5830 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b63fff640 10 monclient: dump: 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: epoch 1 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:27:16.130 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: min_mon_release 19 (squid) 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: election_strategy: 1 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 5 monclient: authenticate success, global_id 4327 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 10 monclient: _renew_subs 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x56266d33dad0 con 0x56266d2a5830 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 10 monclient: _renew_subs 2024-09-10T20:27:16.131 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:27:16.132 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56266d33e210 con 0x56266d2a5830 2024-09-10T20:27:16.132 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 1 librados: init done 2024-09-10T20:27:16.132 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.115+0000 7f7b7d13c880 10 librados: wait_for_osdmap waiting 2024-09-10T20:27:16.132 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b79443640 4 set_mon_vals no callback set 2024-09-10T20:27:16.132 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 <== mon.1 v2:172.21.15.176:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f7b6c0c1990 con 0x56266d2a5830 2024-09-10T20:27:16.133 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.133 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.133 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.133 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.133 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-10T20:27:16.133 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-10T20:27:16.133 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.133 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-10T20:27:16.137 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_stat 2024-09-10T20:27:16.138 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_stat 2024-09-10T20:27:16.138 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 1 --2- 172.21.15.176:0/4265800331 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7b6803b110 0x7f7b6803d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comptest api_stat on pid 28162 2024-09-10T20:27:16.139 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28162 2024-09-10T20:27:16.139 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_stat on pid 28162' 2024-09-10T20:27:16.139 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28162 2024-09-10T20:27:16.139 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.139 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.139 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-10T20:27:16.140 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.140 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.141 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.141 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.141 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-10T20:27:16.141 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_stat: /' 2024-09-10T20:27:16.142 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_stat_pp 2024-09-10T20:27:16.143 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_stat_pp' 2024-09-10T20:27:16.144 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-10T20:27:16.145 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.150 INFO:tasks.workunit.client.0.smithi176.stdout: rx=0 tx=0).connect 2024-09-10T20:27:16.150 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 <== mon.1 v2:172.21.15.176:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3534+0+0 (secure 0 0 0) 0x7f7b6c0a9020 con 0x56266d2a5830 2024-09-10T20:27:16.150 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_map(21..21 src has 1..21) 2024-09-10T20:27:16.150 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 3 client.4327.objecter handle_osd_map got epochs [21,21] > 0 2024-09-10T20:27:16.150 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 3 client.4327.objecter handle_osd_map decoding full epoch 21 2024-09-10T20:27:16.150 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 20 client.4327.objecter dump_active .. 0 homeless 2024-09-10T20:27:16.150 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7ac46640 1 --2- 172.21.15.176:0/4265800331 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7b6803b110 0x7f7b6803d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7d13c880 10 librados: wait_for_osdmap done waiting 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7d13c880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7b6c0b3070 con 0x56266d2a5830 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7ac46640 10 monclient: get_auth_request con 0x7f7b6803b110 auth_method 0 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 <== mon.1 v2:172.21.15.176:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f7b6c0cac50 con 0x56266d2a5830 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 10 monclient: handle_get_version_reply finishing 1 version 21 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b78c42640 10 client.4327.objecter _get_latest_version latest 21, have it 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7d13c880 10 client.4327.objecter create_pool name=ceph_test_rados_api_asio 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7d13c880 10 client.4327.objecter pool_op_submit 1 2024-09-10T20:27:16.151 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7d13c880 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:27:16.152 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x56266d238880 con 0x56266d2a5830 2024-09-10T20:27:16.152 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b7ac46640 1 --2- 172.21.15.176:0/4265800331 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7b6803b110 0x7f7b6803d5d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b64002ac0 tx=0x7f7b64005d30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:27:16.152 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.119+0000 7f7b63fff640 10 client.4327.objecter ms_handle_connect 0x7f7b6803b110 2024-09-10T20:27:16.152 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 <== mon.1 v2:172.21.15.176:3300/0 7 ==== pool_op_reply(tid 1 (0) Success v22) ==== 43+0+0 (secure 0 0 0) 0x7f7b6c0193b0 con 0x56266d2a5830 2024-09-10T20:27:16.152 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 10 client.4327.objecter ms_dispatch 0x56266d207f00 pool_op_reply(tid 1 (0) Success v22) 2024-09-10T20:27:16.152 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 10 client.4327.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v22) 2024-09-10T20:27:16.152 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 10 client.4327.objecter have request 1 at 0x56266d282da0 Op: create 2024-09-10T20:27:16.154 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 20 client.4327.objecter waiting for client to reach epoch 22 before calling back 2024-09-10T20:27:16.154 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:27:16.154 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 10 monclient: _renew_subs 2024-09-10T20:27:16.154 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:27:16.154 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({osdmap=22}) -- 0x7f7b68040530 con 0x56266d2a5830 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.143+0000 7f7b63fff640 10 client.4327.objecter done 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 <== mon.1 v2:172.21.15.176:3300/0 8 ==== osd_map(22..22 src has 1..22) ==== 1383+0+0 (secure 0 0 0) 0x7f7b6c0ace50 con 0x56266d2a5830 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b63fff640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_map(22..22 src has 1..22) 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b63fff640 3 client.4327.objecter handle_osd_map got epochs [22,22] > 21 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b63fff640 3 client.4327.objecter handle_osd_map decoding incremental epoch 22 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b63fff640 20 client.4327.objecter dump_active .. 0 homeless 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 10 librados: set snap read head -> 1 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 10 librados: writefull oid=exist nspace= 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e03f0 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 22 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 1 --2- 172.21.15.176:0/4265800331 >> v2:172.21.15.153:6804/1715391555 conn(0x56266d33b760 0x56266d3e2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:27:16.155 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:16.156 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 2 osd.1 2024-09-10T20:27:16.156 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:16.156 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 2 2024-09-10T20:27:16.156 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 15 client.4327.objecter _send_op 2 to 5.1 on osd.1 2024-09-10T20:27:16.156 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:2 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x56266d3e3300 con 0x56266d33b760 2024-09-10T20:27:16.156 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:16.156 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:16.156 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_stat_pp 2024-09-10T20:27:16.157 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7b447640 1 --2- 172.21.15.176:0/4265800331 >> v2:172.21.15.153:6804/1715391555 conn(0x56266d33b760 0x56266d3e2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handletest api_stat_pp on pid 28188 2024-09-10T20:27:16.157 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_stat_pp 2024-09-10T20:27:16.157 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28188 2024-09-10T20:27:16.157 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_stat_pp on pid 28188' 2024-09-10T20:27:16.157 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28188 2024-09-10T20:27:16.157 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.157 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.160 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-10T20:27:16.161 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.162 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_watch_notify 2024-09-10T20:27:16.163 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_watch_notify' 2024-09-10T20:27:16.163 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.163 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.163 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.163 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.164 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_watch_notify 2024-09-10T20:27:16.164 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-10T20:27:16.165 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-10T20:27:16.165 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-10T20:27:16.165 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_watch_notify 2024-09-10T20:27:16.166 INFO:tasks.workunit.client.0.smithi176.stdout:test api_watch_notify on pid 28212 2024-09-10T20:27:16.166 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28212 2024-09-10T20:27:16.166 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_watch_notify on pid 28212' 2024-09-10T20:27:16.166 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28212 2024-09-10T20:27:16.166 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.166 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.167 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_watch_notify_pp 2024-09-10T20:27:16.167 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_watch_notify_pp' 2024-09-10T20:27:16.167 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-10T20:27:16.168 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.168 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.168 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.168 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.168 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-10T20:27:16.171 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-10T20:27:16.172 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-10T20:27:16.176 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_watch_notify_pp 2024-09-10T20:27:16.176 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.179 INFO:tasks.workunit.client.0.smithi176.stdout:test api_watch_notify_pp on pid 28243 2024-09-10T20:27:16.179 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_watch_notify_pp 2024-09-10T20:27:16.179 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28243 2024-09-10T20:27:16.180 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_watch_notify_pp on pid 28243' 2024-09-10T20:27:16.180 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28243 2024-09-10T20:27:16.180 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.180 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.180 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-10T20:27:16.181 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.181 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.181 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.181 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.181 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-10T20:27:16.181 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-10T20:27:16.182 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-10T20:27:16.182 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_cmd 2024-09-10T20:27:16.182 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_cmd' 2024-09-10T20:27:16.182 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.183 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_cmd 2024-09-10T20:27:16.186 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_cmd 2024-09-10T20:27:16.187 INFO:tasks.workunit.client.0.smithi176.stdout:test api_cmd on pid 28265 2024-09-10T20:27:16.187 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28265 2024-09-10T20:27:16.187 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_cmd on pid 28265' 2024-09-10T20:27:16.187 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28265 2024-09-10T20:27:16.187 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.187 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.188 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-10T20:27:16.188 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.188 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.188 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.188 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.189 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_cmd_pp 2024-09-10T20:27:16.189 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_cmd_pp' 2024-09-10T20:27:16.189 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_cmd_pp 2024-09-10T20:27:16.191 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-10T20:27:16.191 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-10T20:27:16.192 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_cmd: /' 2024-09-10T20:27:16.196 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.198 INFO:tasks.workunit.client.0.smithi176.stdout:test api_cmd_pp on pid 28290 2024-09-10T20:27:16.199 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_cmd_pp 2024-09-10T20:27:16.199 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28290 2024-09-10T20:27:16.199 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_cmd_pp on pid 28290' 2024-09-10T20:27:16.199 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28290 2024-09-10T20:27:16.199 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.199 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.199 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-10T20:27:16.199 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_service 2024-09-10T20:27:16.200 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_service' 2024-09-10T20:27:16.200 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.200 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.200 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.201 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.201 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_service 2024-09-10T20:27:16.201 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-10T20:27:16.201 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-10T20:27:16.205 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-10T20:27:16.213 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.215 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_service 2024-09-10T20:27:16.216 INFO:tasks.workunit.client.0.smithi176.stdout:test api_service on pid 28313 2024-09-10T20:27:16.216 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28313 2024-09-10T20:27:16.216 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_service on pid 28313' 2024-09-10T20:27:16.216 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28313 2024-09-10T20:27:16.216 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.216 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.217 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-10T20:27:16.217 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.217 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.217 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.217 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.218 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-10T20:27:16.218 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_service.log 2024-09-10T20:27:16.218 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_service: /' 2024-09-10T20:27:16.223 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_service_pp 2024-09-10T20:27:16.223 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_service_pp' 2024-09-10T20:27:16.224 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_service_pp 2024-09-10T20:27:16.224 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.227 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_service_pp 2024-09-10T20:27:16.228 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28367 2024-09-10T20:27:16.228 INFO:tasks.workunit.client.0.smithi176.stdout:test api_service_pp on pid 28367 2024-09-10T20:27:16.229 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_service_pp on pid 28367' 2024-09-10T20:27:16.229 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28367 2024-09-10T20:27:16.229 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.229 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.229 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_c_write_operations 2024-09-10T20:27:16.229 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_c_write_operations' 2024-09-10T20:27:16.229 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_c_write_operations 2024-09-10T20:27:16.229 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.232 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_c_write_operations 2024-09-10T20:27:16.232 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28373 2024-09-10T20:27:16.232 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_c_write_operations on pid 28373' 2024-09-10T20:27:16.232 INFO:tasks.workunit.client.0.smithi176.stdout:test api_c_write_operations on pid 28373 2024-09-10T20:27:16.233 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28373 2024-09-10T20:27:16.233 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.233 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.233 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-10T20:27:16.234 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.234 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.234 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.234 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.234 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-10T20:27:16.234 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-10T20:27:16.234 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-10T20:27:16.235 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-10T20:27:16.235 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s api_c_read_operations 2024-09-10T20:27:16.235 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' api_c_read_operations' 2024-09-10T20:27:16.236 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.236 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.236 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.236 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.236 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.237 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-10T20:27:16.237 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-10T20:27:16.237 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo api_c_read_operations 2024-09-10T20:27:16.238 INFO:tasks.workunit.client.0.smithi176.stdout:test api_c_read_operations on pid 28407 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=api_c_read_operations 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28407 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test api_c_read_operations on pid 28407' 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28407 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s list_parallel 2024-09-10T20:27:16.239 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' list_parallel' 2024-09-10T20:27:16.240 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo list_parallel 2024-09-10T20:27:16.241 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.241 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.241 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.241 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.241 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.241 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-10T20:27:16.241 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-10T20:27:16.241 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-10T20:27:16.242 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=list_parallel 2024-09-10T20:27:16.242 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28415 2024-09-10T20:27:16.242 INFO:tasks.workunit.client.0.smithi176.stdout:test list_parallel on pid 28415 2024-09-10T20:27:16.243 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test list_parallel on pid 28415' 2024-09-10T20:27:16.243 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28415 2024-09-10T20:27:16.243 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s open_pools_parallel 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' open_pools_parallel' 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo open_pools_parallel 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-10T20:27:16.244 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-10T20:27:16.246 INFO:tasks.workunit.client.0.smithi176.stdout:test open_pools_parallel on pid 28432 2024-09-10T20:27:16.247 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=open_pools_parallel 2024-09-10T20:27:16.247 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28432 2024-09-10T20:27:16.247 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test open_pools_parallel on pid 28432' 2024-09-10T20:27:16.247 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28432 2024-09-10T20:27:16.247 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-10T20:27:16.247 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.247 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s delete_pools_parallel 2024-09-10T20:27:16.247 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' delete_pools_parallel' 2024-09-10T20:27:16.248 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo delete_pools_parallel 2024-09-10T20:27:16.249 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ list_parallel: /' 2024-09-10T20:27:16.250 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-10T20:27:16.250 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.251 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.251 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.251 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.251 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.251 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-10T20:27:16.252 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-10T20:27:16.253 INFO:tasks.workunit.client.0.smithi176.stdout:test delete_pools_parallel on pid 28453 2024-09-10T20:27:16.254 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=delete_pools_parallel 2024-09-10T20:27:16.254 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28453 2024-09-10T20:27:16.254 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test delete_pools_parallel on pid 28453' 2024-09-10T20:27:16.254 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28453 2024-09-10T20:27:16.254 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.254 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.254 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s cls 2024-09-10T20:27:16.254 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' cls' 2024-09-10T20:27:16.255 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-10T20:27:16.255 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.255 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.256 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.256 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.256 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-10T20:27:16.257 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-10T20:27:16.258 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-10T20:27:16.263 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.263 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo cls 2024-09-10T20:27:16.267 INFO:tasks.workunit.client.0.smithi176.stdout:test cls on pid 28465 2024-09-10T20:27:16.268 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=cls 2024-09-10T20:27:16.268 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28465 2024-09-10T20:27:16.268 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test cls on pid 28465' 2024-09-10T20:27:16.268 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28465 2024-09-10T20:27:16.268 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.268 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.273 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-10T20:27:16.274 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s cmd 2024-09-10T20:27:16.274 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' cmd' 2024-09-10T20:27:16.274 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-10T20:27:16.276 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.276 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.276 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.276 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.276 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_cls.log 2024-09-10T20:27:16.285 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.289 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_cls 2024-09-10T20:27:16.293 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ cls: /' 2024-09-10T20:27:16.294 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo cmd 2024-09-10T20:27:16.295 INFO:tasks.workunit.client.0.smithi176.stdout:test cmd on pid 28495 2024-09-10T20:27:16.296 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=cmd 2024-09-10T20:27:16.296 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28495 2024-09-10T20:27:16.296 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test cmd on pid 28495' 2024-09-10T20:27:16.296 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28495 2024-09-10T20:27:16.296 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.296 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.302 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s handler_error 2024-09-10T20:27:16.303 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' handler_error' 2024-09-10T20:27:16.303 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-10T20:27:16.304 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.305 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.305 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.305 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.305 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-10T20:27:16.306 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ cmd: /' 2024-09-10T20:27:16.307 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_cmd 2024-09-10T20:27:16.310 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo handler_error 2024-09-10T20:27:16.313 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.321 INFO:tasks.workunit.client.0.smithi176.stdout:test handler_error on pid 28591 2024-09-10T20:27:16.321 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=handler_error 2024-09-10T20:27:16.321 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28591 2024-09-10T20:27:16.322 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test handler_error on pid 28591' 2024-09-10T20:27:16.322 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28591 2024-09-10T20:27:16.322 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.322 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.322 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s io 2024-09-10T20:27:16.322 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' io' 2024-09-10T20:27:16.322 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-10T20:27:16.324 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.324 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.325 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.325 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.325 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_handler_error 2024-09-10T20:27:16.325 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-10T20:27:16.325 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ handler_error: /' 2024-09-10T20:27:16.325 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo io 2024-09-10T20:27:16.325 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.327 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=io 2024-09-10T20:27:16.327 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28621 2024-09-10T20:27:16.328 INFO:tasks.workunit.client.0.smithi176.stdout:test io on pid 28621 2024-09-10T20:27:16.328 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test io on pid 28621' 2024-09-10T20:27:16.329 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28621 2024-09-10T20:27:16.329 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-10T20:27:16.329 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.329 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.329 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s ec_io 2024-09-10T20:27:16.329 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' ec_io' 2024-09-10T20:27:16.330 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo ec_io 2024-09-10T20:27:16.330 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.330 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.330 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.330 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.330 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.330 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_io 2024-09-10T20:27:16.330 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_io.log 2024-09-10T20:27:16.335 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ io: /' 2024-09-10T20:27:16.336 INFO:tasks.workunit.client.0.smithi176.stdout:test ec_io on pid 28659 2024-09-10T20:27:16.337 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=ec_io 2024-09-10T20:27:16.337 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28659 2024-09-10T20:27:16.337 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test ec_io on pid 28659' 2024-09-10T20:27:16.337 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28659 2024-09-10T20:27:16.337 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.337 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.337 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-10T20:27:16.338 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s list 2024-09-10T20:27:16.338 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' list' 2024-09-10T20:27:16.338 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.338 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.338 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.338 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.338 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_ec_io 2024-09-10T20:27:16.339 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.339 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ ec_io: /' 2024-09-10T20:27:16.339 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-10T20:27:16.339 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo list 2024-09-10T20:27:16.340 INFO:tasks.workunit.client.0.smithi176.stdout:test list on pid 28668 2024-09-10T20:27:16.341 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=list 2024-09-10T20:27:16.341 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28668 2024-09-10T20:27:16.341 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test list on pid 28668' 2024-09-10T20:27:16.341 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28668 2024-09-10T20:27:16.341 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.341 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.341 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s ec_list 2024-09-10T20:27:16.342 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' ec_list' 2024-09-10T20:27:16.342 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo ec_list 2024-09-10T20:27:16.346 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.349 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=ec_list 2024-09-10T20:27:16.349 INFO:tasks.workunit.client.0.smithi176.stdout:test ec_list on pid 28696 2024-09-10T20:27:16.350 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28696 2024-09-10T20:27:16.350 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test ec_list on pid 28696' 2024-09-10T20:27:16.350 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-10T20:27:16.350 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28696 2024-09-10T20:27:16.350 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.350 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.350 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-10T20:27:16.351 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s misc 2024-09-10T20:27:16.351 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' misc' 2024-09-10T20:27:16.352 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.352 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.352 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.352 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.352 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_list 2024-09-10T20:27:16.352 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_list.log 2024-09-10T20:27:16.352 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ list: /' 2024-09-10T20:27:16.352 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.352 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.353 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.353 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.353 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-10T20:27:16.353 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_ec_list 2024-09-10T20:27:16.353 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.354 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ ec_list: /' 2024-09-10T20:27:16.361 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo misc 2024-09-10T20:27:16.362 INFO:tasks.workunit.client.0.smithi176.stdout:test misc on pid 28728 2024-09-10T20:27:16.362 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=misc 2024-09-10T20:27:16.363 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28728 2024-09-10T20:27:16.363 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test misc on pid 28728' 2024-09-10T20:27:16.363 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28728 2024-09-10T20:27:16.363 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.363 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.363 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-10T20:27:16.364 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.364 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.364 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.364 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.364 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_misc 2024-09-10T20:27:16.365 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_misc.log 2024-09-10T20:27:16.365 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s pool 2024-09-10T20:27:16.365 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' pool' 2024-09-10T20:27:16.366 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ misc: /' 2024-09-10T20:27:16.374 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.375 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo pool 2024-09-10T20:27:16.378 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=pool 2024-09-10T20:27:16.378 INFO:tasks.workunit.client.0.smithi176.stdout:test pool on pid 28783 2024-09-10T20:27:16.379 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28783 2024-09-10T20:27:16.379 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test pool on pid 28783' 2024-09-10T20:27:16.379 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28783 2024-09-10T20:27:16.379 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.379 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.379 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s read_operations 2024-09-10T20:27:16.379 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-10T20:27:16.379 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' read_operations' 2024-09-10T20:27:16.380 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.380 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.380 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.380 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.380 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_pool 2024-09-10T20:27:16.381 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_pool.log 2024-09-10T20:27:16.382 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ pool: /' 2024-09-10T20:27:16.382 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.383 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo read_operations 2024-09-10T20:27:16.386 INFO:tasks.workunit.client.0.smithi176.stdout:test read_operations on pid 28805 2024-09-10T20:27:16.386 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=read_operations 2024-09-10T20:27:16.386 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28805 2024-09-10T20:27:16.386 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test read_operations on pid 28805' 2024-09-10T20:27:16.386 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28805 2024-09-10T20:27:16.386 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.386 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.387 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-10T20:27:16.387 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s snapshots 2024-09-10T20:27:16.387 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' snapshots' 2024-09-10T20:27:16.387 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo snapshots 2024-09-10T20:27:16.387 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.387 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.388 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.388 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.388 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_read_operations 2024-09-10T20:27:16.388 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.389 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=snapshots 2024-09-10T20:27:16.389 INFO:tasks.workunit.client.0.smithi176.stdout:test snapshots on pid 28821 2024-09-10T20:27:16.389 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28821 2024-09-10T20:27:16.390 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test snapshots on pid 28821' 2024-09-10T20:27:16.390 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28821 2024-09-10T20:27:16.390 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.390 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.390 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-10T20:27:16.391 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.391 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.391 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.391 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.391 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s watch_notify 2024-09-10T20:27:16.392 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' watch_notify' 2024-09-10T20:27:16.392 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ snapshots: /' 2024-09-10T20:27:16.392 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ read_operations: /' 2024-09-10T20:27:16.392 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_snapshots 2024-09-10T20:27:16.393 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-10T20:27:16.399 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.399 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-10T20:27:16.402 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo watch_notify 2024-09-10T20:27:16.404 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=watch_notify 2024-09-10T20:27:16.405 INFO:tasks.workunit.client.0.smithi176.stdout:test watch_notify on pid 28860 2024-09-10T20:27:16.405 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28860 2024-09-10T20:27:16.405 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test watch_notify on pid 28860' 2024-09-10T20:27:16.405 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28860 2024-09-10T20:27:16.405 INFO:tasks.workunit.client.0.smithi176.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-10T20:27:16.405 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.405 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-10T20:27:16.406 INFO:tasks.workunit.client.0.smithi176.stderr:++ printf %25s write_operations 2024-09-10T20:27:16.406 INFO:tasks.workunit.client.0.smithi176.stderr:+ r=' write_operations' 2024-09-10T20:27:16.406 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.406 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.406 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.407 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.407 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_watch_notify 2024-09-10T20:27:16.407 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-10T20:27:16.408 INFO:tasks.workunit.client.0.smithi176.stderr:++ echo write_operations 2024-09-10T20:27:16.409 INFO:tasks.workunit.client.0.smithi176.stderr:++ awk '{print $1}' 2024-09-10T20:27:16.414 INFO:tasks.workunit.client.0.smithi176.stdout:test write_operations on pid 28882 2024-09-10T20:27:16.783 INFO:tasks.workunit.client.0.smithi176.stderr:+ ff=write_operations 2024-09-10T20:27:16.783 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ watch_notify: /' 2024-09-10T20:27:16.783 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28882 2024-09-10T20:27:16.783 INFO:tasks.workunit.client.0.smithi176.stderr:+ echo 'test write_operations on pid 28882' 2024-09-10T20:27:16.783 INFO:tasks.workunit.client.0.smithi176.stderr:+ pids[$f]=28882 2024-09-10T20:27:16.783 INFO:tasks.workunit.client.0.smithi176.stderr:+ ret=0 2024-09-10T20:27:16.783 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' 1 -eq 1 ']' 2024-09-10T20:27:16.783 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27960 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 27960 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ '[' -z '' ']' 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ case $- in 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ return 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ ceph_test_neorados_write_operations 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ sed 's/^/ write_operations: /' 2024-09-10T20:27:16.784 INFO:tasks.workunit.client.0.smithi176.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-10T20:27:17.324 INFO:tasks.workunit.client.0.smithi176.stdout:_peer_banner_payload supported=3 required=0 2024-09-10T20:27:17.324 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.147+0000 7f7b7b447640 10 monclient: get_auth_request con 0x56266d33b760 auth_method 0 2024-09-10T20:27:17.324 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.155+0000 7f7b7b447640 1 --2- 172.21.15.176:0/4265800331 >> v2:172.21.15.153:6804/1715391555 conn(0x56266d33b760 0x56266d3e2dc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:27:17.324 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:16.155+0000 7f7b63fff640 10 client.4327.objecter ms_handle_connect 0x56266d33b760 2024-09-10T20:27:17.324 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.295+0000 7f7b63fff640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 1 ==== osd_map(23..23 src has 1..23) ==== 12737+0+0 (crc 0 0 0) 0x7f7b7000e8a0 con 0x56266d33b760 2024-09-10T20:27:17.324 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.295+0000 7f7b63fff640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_map(23..23 src has 1..23) 2024-09-10T20:27:17.324 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.295+0000 7f7b63fff640 3 client.4327.objecter handle_osd_map got epochs [23,23] > 22 2024-09-10T20:27:17.324 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.295+0000 7f7b63fff640 3 client.4327.objecter handle_osd_map decoding incremental epoch 23 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.295+0000 7f7b63fff640 10 client.4327.objecter checking op 2 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.295+0000 7f7b63fff640 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.295+0000 7f7b63fff640 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.295+0000 7f7b63fff640 20 client.4327.objecter dump_active .. 0 homeless 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.295+0000 7f7b63fff640 20 client.4327.objecter 2 5.d3a59201 osd.1 exist [writefull 0~5 in=5b] 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 2 ==== osd_op_reply(2 exist [writefull 0~5] v23'1 uv1 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(2 exist [writefull 0~5] v23'1 uv1 ondisk = 0) 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 2 ondisk uv 1 in 5.1 attempt 0 2024-09-10T20:27:17.325 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 2 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 15 client.4327.objecter _finish_op 2 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 2 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 librados: Objecter returned from writefull r=0 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3df3e0 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.326 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.327 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.327 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.327 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 3 osd.1 2024-09-10T20:27:17.327 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.327 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 3 2024-09-10T20:27:17.327 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 15 client.4327.objecter _send_op 3 to 5.1 on osd.1 2024-09-10T20:27:17.327 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:3 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3df890 con 0x56266d33b760 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3dfc80 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 4 osd.1 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 4 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 15 client.4327.objecter _send_op 4 to 5.1 on osd.1 2024-09-10T20:27:17.328 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:4 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e0130 con 0x56266d33b760 2024-09-10T20:27:17.329 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.329 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 5 client.4327.objecter 2 in flight 2024-09-10T20:27:17.329 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.329 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-10T20:27:17.329 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.329 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 3 ondisk uv 1 in 5.1 attempt 0 2024-09-10T20:27:17.329 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 5 2024-09-10T20:27:17.329 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 3 2024-09-10T20:27:17.329 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 15 client.4327.objecter _finish_op 3 2024-09-10T20:27:17.330 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.330 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 3 2024-09-10T20:27:17.330 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.330 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.330 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-10T20:27:17.330 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.330 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 4 ondisk uv 0 in 5.1 attempt 0 2024-09-10T20:27:17.330 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-10T20:27:17.330 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 4 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 15 client.4327.objecter _finish_op 4 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 4 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (2 ms) 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3dfc80 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.331 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 5 osd.1 2024-09-10T20:27:17.332 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.332 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 5 2024-09-10T20:27:17.332 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 15 client.4327.objecter _send_op 5 to 5.1 on osd.1 2024-09-10T20:27:17.332 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:5 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3df3e0 con 0x56266d33b760 2024-09-10T20:27:17.332 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.332 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.332 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e36a0 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 6 osd.1 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 6 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 15 client.4327.objecter _send_op 6 to 5.1 on osd.1 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:6 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e3b50 con 0x56266d33b760 2024-09-10T20:27:17.333 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.315+0000 7f7b7d13c880 5 client.4327.objecter 2 in flight 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 5 ondisk uv 1 in 5.1 attempt 0 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 5 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 5 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter _finish_op 5 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 5 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.334 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 6 ondisk uv 0 in 5.1 attempt 0 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 6 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter _finish_op 6 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 6 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (5 ms) 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-10T20:27:17.335 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e36a0 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [read 0~256] tid 7 osd.1 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 7 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 15 client.4327.objecter _send_op 7 to 5.1 on osd.1 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:7 5.1 5:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3dfc80 con 0x56266d33b760 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.336 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d423f10 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid noexist '@5' '@5' [read 0~256] tid 8 osd.1 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 8 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 15 client.4327.objecter _send_op 8 to 5.1 on osd.1 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:8 5.1 5:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d4243c0 con 0x56266d33b760 2024-09-10T20:27:17.337 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 5 client.4327.objecter 2 in flight 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 7 ondisk uv 1 in 5.1 attempt 0 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 5 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 7 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter _finish_op 7 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 7 2024-09-10T20:27:17.338 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 8 ondisk uv 0 in 5.1 attempt 0 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 8 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter _finish_op 8 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 8 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.339 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (0 ms) 2024-09-10T20:27:17.340 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-10T20:27:17.340 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-10T20:27:17.340 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 librados: queue_aio_write 0x56266d1fd530 completion 0x56266d3e06b0 write_seq 1 2024-09-10T20:27:17.340 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e36a0 2024-09-10T20:27:17.340 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:27:17.340 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.342 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.342 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.342 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 9 osd.1 2024-09-10T20:27:17.342 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.343 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 9 2024-09-10T20:27:17.343 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 15 client.4327.objecter _send_op 9 to 5.1 on osd.1 2024-09-10T20:27:17.343 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:9 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e4f00 con 0x56266d33b760 2024-09-10T20:27:17.343 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.343 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.343 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.323+0000 7f7b7d13c880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-10T20:27:17.343 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 9 ==== osd_op_reply(9 exist [write 0~5] v23'2 uv2 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.343 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(9 exist [write 0~5] v23'2 uv2 ondisk = 0) 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 9 ondisk uv 2 in 5.1 attempt 0 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 9 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 15 client.4327.objecter _finish_op 9 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (3 ms) 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 9 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.344 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7b447640 20 librados: complete_aio_write 0x56266d3e06b0 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 20 librados: queue_aio_write 0x56266d1fd530 completion 0x56266d3e52a0 write_seq 2 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e36a0 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 10 osd.1 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 10 2024-09-10T20:27:17.345 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 15 client.4327.objecter _send_op 10 to 5.1 on osd.1 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:10 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e4f00 con 0x56266d33b760 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.327+0000 7f7b7d13c880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 10 ==== osd_op_reply(10 exist [write 0~5] v23'3 uv3 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(10 exist [write 0~5] v23'3 uv3 ondisk = 0) 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 10 ondisk uv 3 in 5.1 attempt 0 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 10 2024-09-10T20:27:17.346 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _finish_op 10 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 10 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 20 librados: complete_aio_write 0x56266d3e52a0 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (6 ms) 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 librados: queue_aio_write 0x56266d1fd530 completion 0x56266d3e36a0 write_seq 3 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d427230 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.347 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [write 0~5 in=5b] tid 11 osd.1 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 11 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _send_op 11 to 5.1 on osd.1 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:11 5.1 5:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e37b0 con 0x56266d33b760 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 11 ==== osd_op_reply(11 exist [write 0~5] v23'4 uv4 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.348 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(11 exist [write 0~5] v23'4 uv4 ondisk = 0) 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 11 ondisk uv 4 in 5.1 attempt 0 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 11 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _finish_op 11 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 11 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 20 librados: complete_aio_write 0x56266d3e36a0 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1 ms) 2024-09-10T20:27:17.349 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e3ef0 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 12 osd.1 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 12 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _send_op 12 to 5.1 on osd.1 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:12 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e37b0 con 0x56266d33b760 2024-09-10T20:27:17.350 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.351 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.351 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e44e0 2024-09-10T20:27:17.351 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.351 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-10T20:27:17.351 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 13 osd.1 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 13 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _send_op 13 to 5.1 on osd.1 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:13 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e49b0 con 0x56266d33b760 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 5 client.4327.objecter 2 in flight 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-10T20:27:17.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 12 ondisk uv 4 in 5.1 attempt 0 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 5 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 12 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _finish_op 12 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 12 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 13 ondisk uv 0 in 5.1 attempt 0 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 13 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _finish_op 13 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 13 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (0 ms) 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-10T20:27:17.354 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e44e0 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 14 osd.1 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 14 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _send_op 14 to 5.1 on osd.1 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:14 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e37b0 con 0x56266d33b760 2024-09-10T20:27:17.355 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e7210 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 15 osd.1 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 15 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _send_op 15 to 5.1 on osd.1 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:15 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e49b0 con 0x56266d33b760 2024-09-10T20:27:17.356 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 5 client.4327.objecter 2 in flight 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 14 ondisk uv 4 in 5.1 attempt 0 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 5 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 14 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _finish_op 14 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 14 2024-09-10T20:27:17.357 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 15 ondisk uv 0 in 5.1 attempt 0 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 15 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _finish_op 15 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 15 2024-09-10T20:27:17.358 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e44e0 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [read 0~0] tid 16 osd.1 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.359 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 16 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _send_op 16 to 5.1 on osd.1 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:16 5.1 5:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e37b0 con 0x56266d33b760 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d4278d0 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base noexist @5 precalc_pgid 0 pgid 0.0 is_read 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target noexist @5 -> pgid 5.a7bbaa1 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.a7bbaa1 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid noexist '@5' '@5' [read 0~0] tid 17 osd.1 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.360 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 17 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 15 client.4327.objecter _send_op 17 to 5.1 on osd.1 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:17 5.1 5:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e49b0 con 0x56266d33b760 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.331+0000 7f7b7d13c880 5 client.4327.objecter 2 in flight 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 16 ondisk uv 4 in 5.1 attempt 0 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 5 2024-09-10T20:27:17.361 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 16 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 15 client.4327.objecter _finish_op 16 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 5 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 16 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 17 ondisk uv 0 in 5.1 attempt 0 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 17 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 15 client.4327.objecter _finish_op 17 2024-09-10T20:27:17.362 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 17 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (0 ms) 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 librados: queue_aio_write 0x56266d1fd530 completion 0x56266d3e37b0 write_seq 4 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e44e0 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.363 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 18 osd.1 2024-09-10T20:27:17.364 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.364 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 18 2024-09-10T20:27:17.364 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 15 client.4327.objecter _send_op 18 to 5.1 on osd.1 2024-09-10T20:27:17.365 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:18 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e4990 con 0x56266d33b760 2024-09-10T20:27:17.365 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.365 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.365 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 18 ==== osd_op_reply(18 exist [writefull 0~5] v23'5 uv5 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.365 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(18 exist [writefull 0~5] v23'5 uv5 ondisk = 0) 2024-09-10T20:27:17.365 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.365 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 18 ondisk uv 5 in 5.1 attempt 0 2024-09-10T20:27:17.365 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.365 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 18 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 15 client.4327.objecter _finish_op 18 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 18 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7b447640 20 librados: complete_aio_write 0x56266d3e37b0 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (2 ms) 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 librados: queue_aio_write 0x56266d1fd530 completion 0x56266d3e38c0 write_seq 5 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e44e0 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:27:17.366 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 19 osd.1 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 19 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 15 client.4327.objecter _send_op 19 to 5.1 on osd.1 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:19 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e4990 con 0x56266d33b760 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.335+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 19 ==== osd_op_reply(19 exist [writefull 0~5] v23'6 uv6 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(19 exist [writefull 0~5] v23'6 uv6 ondisk = 0) 2024-09-10T20:27:17.367 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 19 ondisk uv 6 in 5.1 attempt 0 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 19 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 15 client.4327.objecter _finish_op 19 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 19 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7b447640 20 librados: complete_aio_write 0x56266d3e38c0 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (6 ms) 2024-09-10T20:27:17.368 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7d13c880 20 librados: queue_aio_write 0x56266d1fd530 completion 0x56266d3e39d0 write_seq 6 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7d13c880 10 client.4327.objecter _op_submit op 0x56266d3e44e0 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7d13c880 20 client.4327.objecter _calc_target epoch 23 base exist @5 precalc_pgid 0 pgid 0.0 is_write 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7d13c880 20 client.4327.objecter _calc_target target exist @5 -> pgid 5.d3a59201 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7d13c880 10 client.4327.objecter _calc_target raw pgid 5.d3a59201 -> actual 5.1 acting [1,4] primary 1 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7d13c880 20 client.4327.objecter _get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7d13c880 10 client.4327.objecter _op_submit oid exist '@5' '@5' [writefull 0~5 in=5b] tid 20 osd.1 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7d13c880 20 client.4327.objecter get_session s=0x56266d33dd70 osd=1 3 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.339+0000 7f7b7d13c880 15 client.4327.objecter _session_op_assign 1 20 2024-09-10T20:27:17.369 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 15 client.4327.objecter _send_op 20 to 5.1 on osd.1 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 --> v2:172.21.15.153:6804/1715391555 -- osd_op(unknown.0.0:20 5.1 5:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e23) -- 0x56266d3e4990 con 0x56266d33b760 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 5 client.4327.objecter 1 in flight 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 <== osd.1 v2:172.21.15.153:6804/1715391555 20 ==== osd_op_reply(20 exist [writefull 0~5] v23'7 uv7 ondisk = 0) ==== 149+0+0 (crc 0 0 0) 0x7f7b70008030 con 0x56266d33b760 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 10 client.4327.objecter ms_dispatch 0x56266d207f00 osd_op_reply(20 exist [writefull 0~5] v23'7 uv7 ondisk = 0) 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 10 client.4327.objecter in handle_osd_op_reply 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 7 client.4327.objecter handle_osd_op_reply 20 ondisk uv 7 in 5.1 attempt 0 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 10 client.4327.objecter op 0 rval 0 len 0 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 15 client.4327.objecter handle_osd_op_reply completed tid 20 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 15 client.4327.objecter _finish_op 20 2024-09-10T20:27:17.370 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 4 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 15 client.4327.objecter _session_op_remove 1 20 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 5 client.4327.objecter 0 in flight 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 20 librados: complete_aio_write 0x56266d3e39d0 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (2 ms) 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 10 librados: watch_flush enter 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 10 librados: watch_flush exit 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 10 client.4327.objecter close_session for osd.1 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 >> v2:172.21.15.153:6804/1715391555 conn(0x56266d33b760 msgr2=0x56266d3e2dc0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 --2- 172.21.15.176:0/4265800331 >> v2:172.21.15.153:6804/1715391555 conn(0x56266d33b760 0x56266d3e2dc0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 20 client.4327.objecter put_session s=0x56266d33dd70 osd=1 1 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 20 client.4327.objecter shutdown clearing up homeless session... 2024-09-10T20:27:17.371 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 10 client.4327.objecter successfully canceled tick 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7b6803b110 msgr2=0x7f7b6803d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 --2- 172.21.15.176:0/4265800331 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7b6803b110 0x7f7b6803d5d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f7b64002ac0 tx=0x7f7b64005d30 comp rx=0 tx=0).stop 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 10 monclient: shutdown 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d2a5830 msgr2=0x56266d33ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 --2- 172.21.15.176:0/4265800331 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x56266d2a5830 0x56266d33ace0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c025c30 tx=0x7f7b6c0ae5e0 comp rx=0 tx=0).stop 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7b447640 1 -- 172.21.15.176:0/4265800331 reap_dead start 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 shutdown_connections 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 >> 172.21.15.176:0/4265800331 conn(0x56266d28a370 msgr2=0x56266d27e7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 shutdown_connections 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 -- 172.21.15.176:0/4265800331 wait complete. 2024-09-10T20:27:17.372 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.343+0000 7f7b7d13c880 1 librados: shutdown 2024-09-10T20:27:17.373 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [----------] 12 tests from AsioRados (28 ms total) 2024-09-10T20:27:17.373 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: 2024-09-10T20:27:17.373 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [----------] Global test environment tear-down 2024-09-10T20:27:17.373 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1248 ms total) 2024-09-10T20:27:17.373 INFO:tasks.workunit.client.0.smithi176.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-10T20:27:17.373 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:27:17.373 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28102 2024-09-10T20:27:17.373 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28102 2024-09-10T20:27:18.984 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-10T20:27:18.984 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [----------] Global test environment set-up. 2024-09-10T20:27:18.984 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-10T20:27:18.984 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-10T20:27:18.984 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosList.ListObjects (340 ms) 2024-09-10T20:27:18.984 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-10T20:27:18.984 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (39 ms) 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: myset foo1,foo2,foo3 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo1 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo2 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo3 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: myset foo1,foo4,foo5 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo4 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo5 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo1 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: myset foo6,foo7 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo7 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo6 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-10T20:27:18.985 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns1:foo4 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns1:foo5 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns2:foo7 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns2:foo6 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns1:foo1 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: :foo1 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: :foo2 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: :foo3 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (123 ms) 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 1 0 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 10 0 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 13 0 2024-09-10T20:27:18.986 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 7 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 14 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 0 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 15 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 11 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 5 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 8 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 6 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 3 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 4 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 12 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 9 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 2 0 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-10T20:27:18.987 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (124 ms) 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: x cursor=MIN 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=1 cursor=7:02547ec2:::1:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=10 cursor=7:52ea6a34:::10:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=13 cursor=7:566253c9:::13:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=7 cursor=7:5c6b0b28:::7:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=14 cursor=7:62a1935d:::14:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=0 cursor=7:6cac518f:::0:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=15 cursor=7:863748b0:::15:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=11 cursor=7:89d3ae78:::11:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=5 cursor=7:b29083e3:::5:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=8 cursor=7:bd63b0f1:::8:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=6 cursor=7:c4fdafeb:::6:head 2024-09-10T20:27:18.988 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=3 cursor=7:cfc208b3:::3:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=4 cursor=7:d83876eb:::4:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=12 cursor=7:de5d7c5f:::12:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=9 cursor=7:e960b815:::9:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > oid=2 cursor=7:f905c69b:::2:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=1 cursor=7:02547ec2:::1:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=10 cursor=7:52ea6a34:::10:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=13 cursor=7:566253c9:::13:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:566253c9:::13:head 2024-09-10T20:27:18.989 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-09-10T20:27:18.990 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=7 cursor=7:5c6b0b28:::7:head 2024-09-10T20:27:18.990 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-09-10T20:27:18.990 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-09-10T20:27:18.990 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=14 cursor=7:62a1935d:::14:head 2024-09-10T20:27:19.108 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-09-10T20:27:19.108 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-09-10T20:27:19.108 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=0 cursor=7:6cac518f:::0:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=15 cursor=7:863748b0:::15:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:863748b0:::15:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=11 cursor=7:89d3ae78:::11:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=5 cursor=7:b29083e3:::5:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=8 cursor=7:bd63b0f1:::8:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-09-10T20:27:19.109 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=6 cursor=7:c4fdafeb:::6:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=3 cursor=7:cfc208b3:::3:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=4 cursor=7:d83876eb:::4:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=12 cursor=7:de5d7c5f:::12:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=9 cursor=7:e960b815:::9:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:e960b815:::9:head 2024-09-10T20:27:19.110 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : oid=2 cursor=7:f905c69b:::2:head 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:52ea6a34:::10:head 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:52ea6a34:::10:head expected=7:52ea6a34:::10:head 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:52ea6a34:::10:head -> 10 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=10 expected=10 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:566253c9:::13:head 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:566253c9:::13:head expected=7:566253c9:::13:head 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:566253c9:::13:head -> 13 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=13 expected=13 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:89d3ae78:::11:head 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:89d3ae78:::11:head expected=7:89d3ae78:::11:head 2024-09-10T20:27:19.111 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:89d3ae78:::11:head -> 11 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=11 expected=11 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:f905c69b:::2:head 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:f905c69b:::2:head expected=7:f905c69b:::2:head 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:f905c69b:::2:head -> 2 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=2 expected=2 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:e960b815:::9:head 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:e960b815:::9:head expected=7:e960b815:::9:head 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:e960b815:::9:head -> 9 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=9 expected=9 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:863748b0:::15:head 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:863748b0:::15:head expected=7:863748b0:::15:head 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:863748b0:::15:head -> 15 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=15 expected=15 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:6cac518f:::0:head 2024-09-10T20:27:19.112 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:6cac518f:::0:head expected=7:6cac518f:::0:head 2024-09-10T20:27:19.113 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:6cac518f:::0:head -> 0 2024-09-10T20:27:19.113 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=0 expected=0 2024-09-10T20:27:19.113 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:62a1935d:::14:head 2024-09-10T20:27:19.113 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:62a1935d:::14:head expected=7:62a1935d:::14:head 2024-09-10T20:27:19.113 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:62a1935d:::14:head -> 14 2024-09-10T20:27:19.113 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=14 expected=14 2024-09-10T20:27:19.113 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:5c6b0b28:::7:head 2024-09-10T20:27:19.246 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-10T20:27:19.246 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-10T20:27:19.246 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-10T20:27:19.246 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (53 ms) 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (134 ms) 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2841 ms) 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (3028 ms total) 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (3028 ms total) 2024-09-10T20:27:19.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (281 ms) 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-10T20:27:19.256 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (18 ms) 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (33 ms) 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (36 ms) 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (10 ms) 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (13 ms) 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (12 ms) 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (13 ms) 2024-09-10T20:27:19.257 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (12 ms) 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (8 ms) 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (14 ms) 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (10 ms) 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (40 ms) 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (23 ms) 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (10 ms) 2024-09-10T20:27:19.258 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-10T20:27:19.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (5 ms) 2024-09-10T20:27:19.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (538 ms total) 2024-09-10T20:27:19.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: 2024-09-10T20:27:19.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-10T20:27:19.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (2995 ms total) 2024-09-10T20:27:19.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-10T20:27:19.293 INFO:tasks.workunit.client.0.smithi176.stdout: cls: Running main() from gmock_main.cc 2024-09-10T20:27:19.293 INFO:tasks.workunit.client.0.smithi176.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-09-10T20:27:19.293 INFO:tasks.workunit.client.0.smithi176.stdout: cls: [----------] Global test environment set-up. 2024-09-10T20:27:19.293 INFO:tasks.workunit.client.0.smithi176.stdout: cls: [----------] 1 test from NeoRadosCls 2024-09-10T20:27:19.293 INFO:tasks.workunit.client.0.smithi176.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-10T20:27:19.293 INFO:tasks.workunit.client.0.smithi176.stdout: cls: [ OK ] NeoRadosCls.DNE (2986 ms) 2024-09-10T20:27:19.293 INFO:tasks.workunit.client.0.smithi176.stdout: cls: [----------] 1 test from NeoRadosCls (2986 ms total) 2024-09-10T20:27:19.294 INFO:tasks.workunit.client.0.smithi176.stdout: cls: 2024-09-10T20:27:19.294 INFO:tasks.workunit.client.0.smithi176.stdout: cls: [----------] Global test environment tear-down 2024-09-10T20:27:19.294 INFO:tasks.workunit.client.0.smithi176.stdout: cls: [==========] 1 test from 1 test suite ran. (2986 ms total) 2024-09-10T20:27:19.294 INFO:tasks.workunit.client.0.smithi176.stdout: cls: [ PASSED ] 1 test. 2024-09-10T20:27:19.351 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (57 ms) 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (66 ms) 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2924 ms) 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.190894+0000 mon.a [INF] from='client.? 172.21.15.176:0/1512473124' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi176-28165-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.352 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.191119+0000 mon.a [INF] from='client.? 172.21.15.176:0/2352077029' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi176-28388-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.191259+0000 mon.a [INF] from='client.? 172.21.15.176:0/3358989716' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi176-28412-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.191372+0000 mon.a [INF] from='client.? 172.21.15.176:0/3033591951' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "TooBigPP_smithi176-27941-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.191525+0000 mon.a [INF] from='client.4333 172.21.15.176:0/2781670319' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosList_smithi176-27973-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.191636+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLock_smithi176-27978-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.191844+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosLockPP_smithi176-27991-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.192064+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosMiscPP_smithi176-28060-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.192185+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi176-28096-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.192286+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshots_smithi176-28111-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.192388+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi176-28205-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:19.353 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.192488+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi176-28295-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:20.245 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.192647+0000 mon.a [INF] from='client.? ' entity='client. handler_error: Running main() from gmock_main.cc 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: [----------] Global test environment set-up. 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (3893 ms) 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: [----------] 1 test from neocls_handler_error (3893 ms total) 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: [----------] Global test environment tear-down 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: [==========] 1 test from 1 test suite ran. (3893 ms total) 2024-09-10T20:27:20.246 INFO:tasks.workunit.client.0.smithi176.stdout: handler_error: [ PASSED ] 1 test. 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout:admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosIo_smithi176-27949-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.192762+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosSnapshotsPP_smithi176-28135-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.192946+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi176-28218-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:18.193069+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi176-28273-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.204472+0000 mon.a [INF] from='client.? 172.21.15.176:0/1698803881' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi176-28249-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi176-28249-1"}]': finished 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.204602+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi176-28663-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi176-28663-1"}]': finished 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.204684+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi176-28705-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi176-28705-1"}]': finished 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.222492+0000 mon.a [INF] from='client.? 172.21.15.176:0/2393620583' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWrite_smithi176-27932-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.345492+0000 mon.b [INF] from='client.? 172.21.15.176:0/2080512779' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.349045+0000 client.admin [INF] onexx 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.349409+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.380525+0000 mon.b [INF] from='client.? 172.21.15.176:0/2051836479' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi176-28205-7"} : dispatch 2024-09-10T20:27:20.270 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.389279+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi176-28205-7"} : dispatch 2024-09-10T20:27:20.271 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.394295+0000 mon.a [INF] from='client.? 172.21.15.176:0/4170360979' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi176-27956-24"} : dispatch 2024-09-10T20:27:20.271 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.396498+0000 mon.a [INF] from='client.? 172.21.15.176:0/4170360979' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi176-27956-24"} : dispatch 2024-09-10T20:27:20.271 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.397952+0000 mon.b [INF] from='client.4626 172.21.15.176:0/2038257963' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi176-27949-16"} : dispatch 2024-09-10T20:27:20.412 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.404122+0000 mon.b [INF] from='client.? 172.21.15.176:0/2051836479' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name list_parallel: process_1_[28461]: starting. 2024-09-10T20:27:20.412 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_1_[28461]: creating pool ceph_test_rados_list_parallel.smithi176-28421 2024-09-10T20:27:20.412 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_1_[28461]: created object 0... 2024-09-10T20:27:20.412 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_1_[28461]: created object 25... 2024-09-10T20:27:20.412 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_1_[28461]: created object 49... 2024-09-10T20:27:20.412 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_1_[28461]: finishing. 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_1_[28461]: shutting down. 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_2_[28462]: starting. 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_2_[28462]: listing objects. 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_2_[28462]: listed object 0... 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_2_[28462]: listed object 25... 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_2_[28462]: saw 50 objects 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_2_[28462]: shutting down. 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_3_[29060]: starting. 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_3_[29060]: creating pool ceph_test_rados_list_parallel.smithi176-28421 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_3_[29060]: created object 0... 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_3_[29060]: created object 25... 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_3_[29060]: created object 49... 2024-09-10T20:27:20.413 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_3_[29060]: finishing. 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_3_[29060]: shutting down. 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_5_[29062]: starting. 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_5_[29062]: removed 25 objects... 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_5_[29062]: removed half of the objects 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_5_[29062]: removed 50 objects... 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_5_[29062]: removed 50 objects 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_5_[29062]: shutting down. 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_4_[29061]: starting. 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_4_[29061]: listing objects. 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_4_[29061]: listed object 0... 2024-09-10T20:27:20.414 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_4_[29061]: saw 4 objects 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_4_[29061]: shutting down. 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_6_[29111]: starting. 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_6_[29111]: creating pool ceph_test_rados_list_parallel.smithi176-28421 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_6_[29111]: created object 0... 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_6_[29111]: created object 25... 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_6_[29111]: created object 49... 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_6_[29111]: finishing. 2024-09-10T20:27:20.415 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_6_[29111]: shutting down. 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_7_[29112]: starting. 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_7_[29112]: listing objects. 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_7_[29112]: listed object 0... 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_7_[29112]: listed object 25... 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_7_[29112]: listed object 50... 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_7_[29112]: saw 55 objects 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_7_[29112]: shutting down. 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_8_[29113]: starting. 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_8_[29113]: added 25 objects... 2024-09-10T20:27:20.416 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_8_[29113]: added half of the objects 2024-09-10T20:27:20.417 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_8_[29113]: added 50 objects... 2024-09-10T20:27:20.417 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_8_[29113]: added 50 objects 2024-09-10T20:27:20.417 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_8_[29113]: shutting down. 2024-09-10T20:27:20.417 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.417 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.417 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:20.417 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_9_[29440]: starting. 2024-09-10T20:27:21.537 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_9_[29440]: creating pool ceph_test_rados_list_parallel.smithi176-28421 2024-09-10T20:27:21.538 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_9_[29440]: created object 0... 2024-09-10T20:27:21.538 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_9_[29440]: created object 25... 2024-09-10T20:27:21.538 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_9_[29440]: created object 49... 2024-09-10T20:27:21.538 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_9_[29440]: finishing. 2024-09-10T20:27:21.538 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_9_[29440]: shutting down. 2024-09-10T20:27:21.538 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.538 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.538 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.538 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_10_[29441]: starting. 2024-09-10T20:27:21.539 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_10_[29441]: listing objects. 2024-09-10T20:27:21.539 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_10_[29441]: listed object 0... 2024-09-10T20:27:21.539 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_10_[29441]: listed object 25... 2024-09-10T20:27:21.539 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_10_[29441]: listed object 50... 2024-09-10T20:27:21.539 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_10_[29441]: listed object 75... 2024-09-10T20:27:21.540 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_10_[29441]: listed object 100... 2024-09-10T20:27:21.540 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_10_[29441]: saw 101 objects 2024-09-10T20:27:21.540 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_10_[29441]: shutting down. 2024-09-10T20:27:21.540 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.540 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.540 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.540 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_13_[29444]: starting. 2024-09-10T20:27:21.541 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_13_[29444]: removed 25 objects... 2024-09-10T20:27:21.541 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_13_[29444]: removed half of the objects 2024-09-10T20:27:21.541 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_13_[29444]: removed 50 objects... 2024-09-10T20:27:21.541 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_13_[29444]: removed 50 objects 2024-09-10T20:27:21.541 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_13_[29444]: shutting down. 2024-09-10T20:27:21.541 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.542 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.542 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.542 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_11_[29442]: starting. 2024-09-10T20:27:21.542 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_11_[29442]: added 25 objects... 2024-09-10T20:27:21.542 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_11_[29442]: added half of the objects 2024-09-10T20:27:21.543 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_11_[29442]: added 50 objects... 2024-09-10T20:27:21.543 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_11_[29442]: added 50 objects 2024-09-10T20:27:21.543 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_11_[29442]: shutting down. 2024-09-10T20:27:21.544 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.544 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.544 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.544 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_12_[29443]: starting. 2024-09-10T20:27:21.544 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_12_[29443]: added 25 objects... 2024-09-10T20:27:21.545 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_12_[29443]: added half of the objects 2024-09-10T20:27:21.545 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_12_[29443]: added 50 objects... 2024-09-10T20:27:21.545 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_12_[29443]: added 50 objects 2024-09-10T20:27:21.545 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_12_[29443]: shutting down. 2024-09-10T20:27:21.546 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.546 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.546 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.546 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.546 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_14_[29564]: starting. 2024-09-10T20:27:21.546 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_14_[29564]: creating pool ceph_test_rados_list_parallel.smithi176-28421 2024-09-10T20:27:21.546 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_14_[29564]: created object 0... 2024-09-10T20:27:21.546 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_14_[29564]: created object 25... 2024-09-10T20:27:21.546 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_14_[29564]: created object 49... 2024-09-10T20:27:21.547 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_14_[29564]: finishing. 2024-09-10T20:27:21.547 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_14_[29564]: shutting down. 2024-09-10T20:27:21.547 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.547 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.547 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.547 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:21.547 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: starting. 2024-09-10T20:27:21.547 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: listing objects. 2024-09-10T20:27:21.547 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: listed object 0... 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: listed object 25... 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: listed object 50... 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: listed object 75... 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: listed object 100... 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: listed object 125... 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: saw 150 objects 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_15_[29565]: shutting down. 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:22.451 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_16_[29566]: starting. 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_16_[29566]: added 25 objects... 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_16_[29566]: added half of the objects 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_16_[29566]: added 50 objects... 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_16_[29566]: added 50 objects 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: process_16_[29566]: shutting down. 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******************************* 2024-09-10T20:27:22.452 INFO:tasks.workunit.client.0.smithi176.stdout: list_parallel: ******* SUCCESS ********** 2024-09-10T20:27:22.464 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_1_[28469]: starting. 2024-09-10T20:27:22.464 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_1_[28469]: creating pool ceph_test_rados_delete_pools_parallel.smithi176-28456 2024-09-10T20:27:22.464 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_1_[28469]: created object 0... 2024-09-10T20:27:22.464 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_1_[28469]: created object 25... 2024-09-10T20:27:22.464 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_1_[28469]: created object 49... 2024-09-10T20:27:22.464 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_1_[28469]: finishing. 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_1_[28469]: shutting down. 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_2_[28470]: starting. 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_2_[28470]: deleting pool ceph_test_rados_delete_pools_parallel.smithi176-28456 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_2_[28470]: shutting down. 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: ******************************* 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_3_[29114]: starting. 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_3_[29114]: creating pool ceph_test_rados_delete_pools_parallel.smithi176-28456 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_3_[29114]: created object 0... 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_3_[29114]: created object 25... 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_3_[29114]: created object 49... 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_3_[29114]: finishing. 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_3_[29114]: shutting down. 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: ******************************* 2024-09-10T20:27:22.465 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_5_[29116]: starting. 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_5_[29116]: listing objects. 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_5_[29116]: listed object 0... 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_5_[29116]: listed object 25... 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_5_[29116]: saw 50 objects 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_5_[29116]: shutting down. 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: ******************************* 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_4_[29115]: starting. 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_4_[29115]: deleting pool ceph_test_rados_delete_pools_parallel.smithi176-28456 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: process_4_[29115]: shutting down. 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: ******************************* 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: ******************************* 2024-09-10T20:27:22.466 INFO:tasks.workunit.client.0.smithi176.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-10T20:27:22.485 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_1_[28463]: starting. 2024-09-10T20:27:22.485 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_1_[28463]: creating pool ceph_test_rados_open_pools_parallel.smithi176-28449 2024-09-10T20:27:22.485 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_1_[28463]: created object 0... 2024-09-10T20:27:22.485 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_1_[28463]: created object 25... 2024-09-10T20:27:22.485 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_1_[28463]: created object 49... 2024-09-10T20:27:22.485 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_1_[28463]: finishing. 2024-09-10T20:27:22.485 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_1_[28463]: shutting down. 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_2_[28464]: starting. 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_2_[28464]: rados_pool_create. 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_2_[28464]: rados_ioctx_create. 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_2_[28464]: shutting down. 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: ******************************* 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_3_[29132]: starting. 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_3_[29132]: creating pool ceph_test_rados_open_pools_parallel.smithi176-28449 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_3_[29132]: created object 0... 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_3_[29132]: created object 25... 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_3_[29132]: created object 49... 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_3_[29132]: finishing. 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_3_[29132]: shutting down. 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: ******************************* 2024-09-10T20:27:22.486 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_4_[29133]: starting. 2024-09-10T20:27:22.487 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_4_[29133]: rados_pool_create. 2024-09-10T20:27:22.487 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_4_[29133]: rados_ioctx_create. 2024-09-10T20:27:22.487 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: process_4_[29133]: shutting down. 2024-09-10T20:27:22.487 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: ******************************* 2024-09-10T20:27:22.487 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: ******************************* 2024-09-10T20:27:22.487 INFO:tasks.workunit.client.0.smithi176.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-10T20:27:23.283 INFO:tasks.workunit.client.0.smithi176.stdout:":"LibRadosStatECPP_smithi176-28205-7"} : dispatch 2024-09-10T20:27:23.283 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.409052+0000 mon.a [INF] from='client.? 172.21.15.176:0/4170360979' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi176-27956-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.283 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.409883+0000 mon.a [INF] from='client.4626 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi176-27949-16"} : dispatch 2024-09-10T20:27:23.283 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.414494+0000 mon.b [INF] from='client.4626 172.21.15.176:0/2038257963' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi176-27949-16"} : dispatch 2024-09-10T20:27:23.283 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.426613+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi176-28205-7"} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.427071+0000 mon.a [INF] from='client.4626 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi176-27949-16"} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.428062+0000 mon.b [INF] from='client.4626 172.21.15.176:0/2038257963' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi176-27949-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.428252+0000 mon.b [INF] from='client.? 172.21.15.176:0/2051836479' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi176-28205-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.431187+0000 mon.a [INF] from='client.4626 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi176-27949-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.431664+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi176-28205-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.434929+0000 mon.a [INF] from='client.? 172.21.15.176:0/3782535197' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi176-28165-7"} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.439101+0000 mon.a [INF] from='client.? 172.21.15.176:0/3782535197' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi176-28165-7"} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:19.439660+0000 mon.a [INF] from='client.? 172.21.15.176:0/3782535197' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi176-28165-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.266163+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi176-28663-1"} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.268905+0000 mon.b [INF] from='client.? 172.21.15.176:0/1689596155' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi176-28705-1"} : dispatch 2024-09-10T20:27:23.284 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.274335+0000 mon.a [INF] from='client.? 172.21.15.176:0/1698803881' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi176-28249-1"} : dispatch 2024-09-10T20:27:23.460 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.274962+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix":"log", "logtext cmd: Running main() from gmock_main.cc 2024-09-10T20:27:23.460 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-10T20:27:23.460 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [----------] Global test environment set-up. 2024-09-10T20:27:23.460 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-10T20:27:23.460 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1900 ms) 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2015 ms) 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3205 ms) 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [----------] 3 tests from NeoRadosCmd (7120 ms total) 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [----------] Global test environment tear-down 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [==========] 3 tests from 1 test suite ran. (7121 ms total) 2024-09-10T20:27:23.461 INFO:tasks.workunit.client.0.smithi176.stdout: cmd: [ PASSED ] 3 tests. 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout:":["twoxx"]}]': finished 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.285133+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi176-28705-1"} : dispatch 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.378802+0000 mon.b [INF] from='client.? 172.21.15.176:0/1139936330' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi176-27978-10"} : dispatch 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.386774+0000 mon.b [INF] from='client.? 172.21.15.176:0/3480575234' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi176-27991-10"} : dispatch 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.387332+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi176-27978-10"} : dispatch 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.389632+0000 mon.b [INF] from='client.? 172.21.15.176:0/1139936330' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi176-27978-10"} : dispatch 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.391429+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi176-27991-10"} : dispatch 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.393926+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi176-27978-10"} : dispatch 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.393955+0000 mon.b [INF] from='client.? 172.21.15.176:0/1139936330' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi176-27978-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.462 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.396122+0000 mon.b [INF] from='client.? 172.21.15.176:0/3480575234' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi176-27991-10"} : dispatch 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.401535+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi176-27978-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.403303+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi176-27991-10"} : dispatch 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.403902+0000 mon.b [INF] from='client.? 172.21.15.176:0/3480575234' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi176-27991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:21.419840+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi176-27991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.395752+0000 mon.a [INF] from='client.? 172.21.15.176:0/4170360979' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi176-27956-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi176-27956-24"}]': finished 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.395888+0000 mon.a [INF] from='client.? 172.21.15.176:0/3782535197' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi176-28165-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi176-28165-7"}]': finished 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.396064+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi176-28205-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi176-28205-7"}]': finished 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.396236+0000 mon.a [INF] from='client.4626 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi176-27949-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi176-27949-16"}]': finished 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.396353+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"SimpleWritesmithi176-28663-1"}]': finished 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.396486+0000 mon.a [INF] from='client.? 172.21.15.176:0/1698803881' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi176-28249-1"}]': finished 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.396583+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi176-28705-1"}]': finished 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.396753+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi176-27978-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.396870+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi176-27991-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T20:27:23.463 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.407770+0000 mon.b [INF] from='client.? 172.21.15.176:0/1689596155' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi176-28705-1"} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.409142+0000 mon.b [INF] from='client.? 172.21.15.176:0/3480575234' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi176-27991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi176-27991-10"} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.411234+0000 mon.b [INF] from='client.? 172.21.15.176:0/1139936330' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi176-27978-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi176-27978-10"} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.413049+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"ListObjectssmithi176-28705-1"} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.413412+0000 mon.b [INF] from='client.? 172.21.15.176:0/3467958228' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi176-27941-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.413792+0000 mon.a [INF] from='client.? 172.21.15.176:0/1698803881' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi176-28249-1"} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.414705+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi176-27991-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi176-27991-10"} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.440966+0000 mon.b [INF] from='client.? 172.21.15.176:0/90747054' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi176-27932-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.484953+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi176-27978-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi176-27978-10"} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.488886+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi176-27941-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-10T20:27:23.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.489246+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "WaitForSafe_smithi176-27932-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.532136+0000 mon.b [INF] from='client.? 172.21.15.176:0/3539126877' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi176-28663-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.572724+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi176-28663-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:22.695526+0000 mon.a [WRN] Health check update: 15 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.280584+0000 mon.b [INF] from='client.? 172.21.15.176:0/2080512779' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.282114+0000 client.admin [INF] threexx 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.282778+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.407290+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi176-28705-1"}]': finished 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.407495+0000 mon.a [INF] from='client.? 172.21.15.176:0/1698803881' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi176-28249-1"}]': finished 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.407678+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi176-27941-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.407811+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi176-27932-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.407962+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi176-28663-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.426805+0000 mon.b [INF] from='client.? 172.21.15.176:0/3467958228' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi176-27941-3", "field": "max_bytes", "val": "4096"} : dispatch 2024-09-10T20:27:23.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.427798+0000 mon.b [INF] from='client.? 172.21.15.176:0/3539126877' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi176-28663-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi176-28663-2"} : dispatch 2024-09-10T20:27:23.608 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.441730+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi176-27941-3", "field api_io_pp: Running main() from gmock_main.cc 2024-09-10T20:27:23.608 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-09-10T20:27:23.608 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: seed 27956 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (88 ms) 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (22 ms) 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (5 ms) 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-09-10T20:27:23.609 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (711 ms) 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (11 ms) 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (8 ms) 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (8 ms) 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (4 ms) 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (15 ms) 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (13 ms) 2024-09-10T20:27:23.610 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (4 ms) 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (11 ms) 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (8 ms) 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (7 ms) 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (8 ms) 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (19 ms) 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-10T20:27:23.611 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (11 ms) 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (7 ms) 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (8 ms) 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (6 ms) 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (276 ms) 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (1251 ms total) 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1136 ms) 2024-09-10T20:27:23.612 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-10T20:27:23.613 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (19 ms) 2024-09-10T20:27:23.613 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-10T20:27:23.613 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (18 ms) 2024-09-10T20:27:26.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: Running main() from gmock_main.cc 2024-09-10T20:27:26.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-10T20:27:26.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [----------] Global test environment set-up. 2024-09-10T20:27:26.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-10T20:27:26.464 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (291 ms) 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: no timeout :/ 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: no timeout :/ 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: no timeout :/ 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: no timeout :/ 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: no timeout :/ 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (57 ms) 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-10T20:27:26.465 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (22 ms) 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.Checksum (5 ms) 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (9 ms) 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (7 ms) 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (5 ms) 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (1 ms) 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.TruncTest (5 ms) 2024-09-10T20:27:26.466 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (6 ms) 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (5 ms) 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.RmXattr (36 ms) 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIo.XattrIter (6 ms) 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [----------] 14 tests from LibRadosIo (456 ms total) 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-10T20:27:26.467 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1135 ms) 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (21 ms) 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (20 ms) 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (12 ms) 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (11 ms) 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (13 ms) 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-10T20:27:26.468 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (7 ms) 2024-09-10T20:27:26.469 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-10T20:27:26.469 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (9 ms) 2024-09-10T20:27:26.469 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-10T20:27:26.469 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (38 ms) 2024-09-10T20:27:26.469 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-10T20:27:26.469 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (14 ms) 2024-09-10T20:27:26.469 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1280 ms total) 2024-09-10T20:27:26.469 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: watch_notify: Running main() from gmock_main.cc 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: handle_notify cookie 94563706461584 notify_id 103079215105 notifier_gid 4486 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2805 ms) 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: Trying... 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: handle_notify cookie 94563719158192 notify_id 115964116994 notifier_gid 4573 2024-09-10T20:27:26.475 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: Waiting for 3.000000000s 2024-09-10T20:27:26.476 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: Timed out. 2024-09-10T20:27:26.476 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: Flushing... 2024-09-10T20:27:26.476 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: Flushed... 2024-09-10T20:27:26.476 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7235 ms) 2024-09-10T20:27:26.476 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (10040 ms total) 2024-09-10T20:27:26.476 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: 2024-09-10T20:27:26.476 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-10T20:27:26.476 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (10040 ms total) 2024-09-10T20:27:26.476 INFO:tasks.workunit.client.0.smithi176.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-10T20:27:26.480 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: seed 28205 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (271 ms) 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (25 ms) 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (4 ms) 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (7 ms) 2024-09-10T20:27:26.481 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (33 ms) 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (340 ms total) 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1146 ms) 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (25 ms) 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (25 ms) 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1197 ms total) 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: 2024-09-10T20:27:26.482 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-10T20:27:26.483 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10285 ms total) 2024-09-10T20:27:26.483 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-10T20:27:26.483 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: Running main() from gmock_main.cc 2024-09-10T20:27:26.483 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [----------] Global test environment set-up. 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ OK ] LibRadosStat.Stat (272 ms) 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (3 ms) 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ OK ] LibRadosStat.StatNS (39 ms) 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (1 ms) 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (14 ms) 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [----------] 5 tests from LibRadosStat (329 ms total) 2024-09-10T20:27:26.484 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1143 ms) 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (37 ms) 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (0 ms) 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (5 ms) 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1185 ms total) 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [----------] Global test environment tear-down 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (10301 ms total) 2024-09-10T20:27:26.485 INFO:tasks.workunit.client.0.smithi176.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-10T20:27:26.486 INFO:tasks.workunit.client.0.smithi176.stdout:[----------] Global test environment tear-down 2024-09-10T20:27:26.486 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [==========] 24 tests from 2 test suites ran. (10404 ms total) 2024-09-10T20:27:26.486 INFO:tasks.workunit.client.0.smithi176.stdout: api_io: [ PASSED ] 24 tests. 2024-09-10T20:27:26.495 INFO:tasks.workunit.client.0.smithi176.stdout: list: Running main() from gmock_main.cc 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [----------] Global test environment set-up. 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [----------] 3 tests from NeoradosList 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [ OK ] NeoradosList.ListObjects (2858 ms) 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3191 ms) 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [ OK ] NeoradosList.ListObjectsMany (4054 ms) 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [----------] 3 tests from NeoradosList (10104 ms total) 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: 2024-09-10T20:27:26.496 INFO:tasks.workunit.client.0.smithi176.stdout: list: [----------] Global test environment tear-down 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: list: [==========] 3 tests from 1 test suite ran. (10105 ms total) 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: list: [ PASSED ] 3 tests. 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout:": "max_bytes", "val": "4096"} : dispatch 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.442104+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi176-28663-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi176-28663-2"} : dispatch 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.445096+0000 mon.b [INF] from='client.? 172.21.15.176:0/2172957927' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi176-28096-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: got: 2024-09-10T20:27:23.447581+0000 mon.a [INF] from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi176-28096-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (7237 ms) 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (10284 ms total) 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (10284 ms total) 2024-09-10T20:27:26.497 INFO:tasks.workunit.client.0.smithi176.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: Running main() from gmock_main.cc 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [----------] Global test environment set-up. 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (6066 ms) 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (6313 ms) 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12379 ms total) 2024-09-10T20:27:28.748 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: 2024-09-10T20:27:28.749 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [----------] Global test environment tear-down 2024-09-10T20:27:28.749 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12379 ms total) 2024-09-10T20:27:28.749 INFO:tasks.workunit.client.0.smithi176.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-10T20:27:28.974 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: notify 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1134 ms) 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (20 ms) 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1154 ms total) 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: 2024-09-10T20:27:28.975 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: notify 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (136 ms) 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: notify 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3283 ms) 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (9 ms) 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (21 ms) 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609889648 notify_id 150323855360 notifier_gid 4852 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (15 ms) 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-10T20:27:28.976 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609906016 notify_id 150323855361 notifier_gid 4852 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (16 ms) 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609906016 notify_id 150323855362 notifier_gid 4852 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (9 ms) 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609889648 notify_id 150323855360 notifier_gid 4852 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609889648 notify_id 150323855361 notifier_gid 4852 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (18 ms) 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609889648 notify_id 150323855362 notifier_gid 4852 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (6 ms) 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-10T20:27:28.977 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609906016 notify_id 150323855363 notifier_gid 4852 2024-09-10T20:27:28.978 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (16 ms) 2024-09-10T20:27:28.978 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-10T20:27:28.978 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: trying... 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609906016 notify_id 150323855364 noti pool: Running main() from gmock_main.cc 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [----------] Global test environment set-up. 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ OK ] NeoRadosPools.PoolList (1836 ms) 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2023 ms) 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-10T20:27:29.755 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2177 ms) 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4041 ms) 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (1263 ms) 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (2012 ms) 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [----------] 6 tests from NeoRadosPools (13352 ms total) 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [----------] Global test environment tear-down 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [==========] 6 tests from 1 test suite ran. (13352 ms total) 2024-09-10T20:27:29.756 INFO:tasks.workunit.client.0.smithi176.stdout: pool: [ PASSED ] 6 tests. 2024-09-10T20:27:30.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-10T20:27:30.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-10T20:27:30.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-10T20:27:30.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-10T20:27:30.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: seed 27991 2024-09-10T20:27:30.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-10T20:27:30.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (243 ms) 2024-09-10T20:27:30.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-10T20:27:30.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (59 ms) 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1042 ms) 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1006 ms) 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (5 ms) 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (11 ms) 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (6 ms) 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-10T20:27:30.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (4 ms) 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2376 ms total) 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1131 ms) 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (12 ms) 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1107 ms) 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1011 ms) 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-10T20:27:30.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (9 ms) 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (100 ms) 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (41 ms) 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (5 ms) 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3417 ms total) 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (14663 ms total) 2024-09-10T20:27:30.775 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-10T20:27:30.806 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: Running main() from gmock_main.cc 2024-09-10T20:27:30.806 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [----------] Global test environment set-up. 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (262 ms) 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLock.LockShared (19 ms) 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1079 ms) 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-10T20:27:30.807 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1016 ms) 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (10 ms) 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (15 ms) 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (3 ms) 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [----------] 8 tests from LibRadosLock (2408 ms total) 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-10T20:27:30.808 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1157 ms) 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (9 ms) 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1090 ms) 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1018 ms) 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (92 ms) 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (9 ms) 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (26 ms) 2024-09-10T20:27:30.809 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-10T20:27:30.810 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (14 ms) 2024-09-10T20:27:30.810 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3416 ms total) 2024-09-10T20:27:30.810 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: 2024-09-10T20:27:30.810 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [----------] Global test environment tear-down 2024-09-10T20:27:30.810 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (14720 ms total) 2024-09-10T20:27:30.810 INFO:tasks.workunit.client.0.smithi176.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-10T20:27:35.766 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-10T20:27:35.766 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify_test_cb 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (321 ms) 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94386093483664 err -107 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (18 ms) 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-10T20:27:35.767 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94386093495200 err -107 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (1016 ms) 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_cb from 4378 notify_id 107374182400 cookie 94386093513184 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (9 ms) 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_cb from 4378 notify_id 107374182400 cookie 94386093531056 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (16 ms) 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_cb from 4378 notify_id 107374182400 cookie 94386093550544 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (11 ms) 2024-09-10T20:27:35.768 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_cb from 4378 notify_id 107374182401 cookie 94386093539504 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_cb from 4378 notify_id 107374182401 cookie 94386093550544 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (24 ms) 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_cb from 4378 notify_id 107374182401 cookie 94386093550544 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_cb from 4378 notify_id 111669149698 cookie 94386093550544 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3010 ms) 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94386093550544 err -107 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_cb from 4378 notify_id 141733920772 cookie 94386093550544 2024-09-10T20:27:35.769 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5012 ms) 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94386093550544 err -107 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1010 ms) 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (10447 ms total) 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: watch_notify_test_cb 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1086 ms) 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1086 ms total) 2024-09-10T20:27:35.770 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: 2024-09-10T20:27:35.771 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-10T20:27:35.771 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (19576 ms total) 2024-09-10T20:27:35.771 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-10T20:27:35.771 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-10T20:27:35.771 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (5 ms) 2024-09-10T20:27:35.771 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-10T20:27:35.771 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (5 ms) 2024-09-10T20:27:35.771 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (18 ms) 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (9 ms) 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (3 ms) 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (6 ms) 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (7 ms) 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-10T20:27:35.772 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (8 ms) 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (6 ms) 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (30 ms) 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (8073 ms) 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1081 ms) 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (11 ms) 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-10T20:27:35.773 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-09-10T20:27:35.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-10T20:27:35.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (11 ms) 2024-09-10T20:27:35.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (10451 ms total) 2024-09-10T20:27:35.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: 2024-09-10T20:27:35.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-10T20:27:35.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (19695 ms total) 2024-09-10T20:27:35.774 INFO:tasks.workunit.client.0.smithi176.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-09-10T20:27:39.777 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: Running main() from gmock_main.cc 2024-09-10T20:27:39.777 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [----------] Global test environment set-up. 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (7057 ms) 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6311 ms) 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (10038 ms) 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [----------] 3 tests from NeoradosECList (23406 ms total) 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [----------] Global test environment tear-down 2024-09-10T20:27:39.778 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (23406 ms total) 2024-09-10T20:27:39.779 INFO:tasks.workunit.client.0.smithi176.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: Running main() from gmock_main.cc 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [----------] Global test environment set-up. 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (2805 ms) 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3225 ms) 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-10T20:27:41.834 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (2999 ms) 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (2267 ms) 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (3051 ms) 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (2967 ms) 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (8074 ms) 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (25388 ms total) 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [----------] Global test environment tear-down 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (25388 ms total) 2024-09-10T20:27:41.835 INFO:tasks.workunit.client.0.smithi176.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (2980 ms) 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-10T20:27:41.838 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3184 ms) 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3030 ms) 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (2261 ms) 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (3066 ms) 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (2947 ms) 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (8097 ms) 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (25565 ms total) 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: 2024-09-10T20:27:41.839 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-10T20:27:41.840 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (25565 ms total) 2024-09-10T20:27:41.840 INFO:tasks.workunit.client.0.smithi176.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-10T20:27:43.800 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [----------] Global test environment set-up. 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5077 ms) 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [ OK ] LibRadosService.RegisterLate (82 ms) 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: cluster: 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: id: 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: health: HEALTH_WARN 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 15 pool(s) do not have an application enabled 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.801 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: services: 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: mon: 3 daemons, quorum a,b (age 33s), out of quorum: c 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: mgr: x(active, since 76s) 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: osd: 6 osds: 6 up (since 69s), 6 in (since 79s) 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: data: 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: pools: 25 pools, 697 pgs 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: objects: 6 objects, 577 KiB 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: usage: 164 MiB used, 540 GiB / 540 GiB avail 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: pgs: 68.436% pgs unknown 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 24.964% pgs not active 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 477 unknown 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 124 creating+activating 2024-09-10T20:27:43.802 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 50 creating+peering 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 43 active+clean 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 3 active 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: io: 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: client: 1.5 KiB/s rd, 2.0 KiB/s wr, 1 op/s rd, 1 op/s wr 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: cluster: 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: id: 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: health: HEALTH_WARN 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 12 pool(s) do not have an application enabled 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: services: 2024-09-10T20:27:43.803 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: mon: 3 daemons, quorum a,b (age 35s), out of quorum: c 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: mgr: x(active, since 78s) 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: osd: 6 osds: 6 up (since 71s), 6 in (since 81s) 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: data: 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: pools: 20 pools, 489 pgs 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: objects: 25 objects, 579 KiB 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: usage: 165 MiB used, 540 GiB / 540 GiB avail 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: pgs: 46.217% pgs unknown 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 20.654% pgs not active 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 226 unknown 2024-09-10T20:27:43.804 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 161 active+clean 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 68 creating+activating 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 33 creating+peering 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 1 active 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2332 ms) 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [ OK ] LibRadosService.Status (20054 ms) 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [----------] 4 tests from LibRadosService (27545 ms total) 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [----------] Global test environment tear-down 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27545 ms total) 2024-09-10T20:27:43.805 INFO:tasks.workunit.client.0.smithi176.stdout: api_service: [ PASSED ] 4 tests. 2024-09-10T20:27:44.795 INFO:tasks.workunit.client.0.smithi176.stdout:fier_gid 4852 2024-09-10T20:27:44.795 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: timed out 2024-09-10T20:27:44.795 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: flushing 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: flushed 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3009 ms) 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: trying... 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609906016 notify_id 163208757253 notifier_gid 4852 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: timed out 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: flushing 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: flushed 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3017 ms) 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: List watches 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: notify2 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609906016 notify_id 176093659142 notifier_gid 4852 2024-09-10T20:27:44.796 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: notify2 done 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: watch_check 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: unwatch2 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: flushing 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: done 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3024 ms) 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: List watches 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: notify2 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: handle_notify cookie 94132609906016 notify_id 188978561031 notifier_gid 4852 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: notify2 done 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: watch_check 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: unwatch2 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: flushing 2024-09-10T20:27:44.797 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: done 2024-09-10T20:27:44.798 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3035 ms) 2024-09-10T20:27:44.798 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (15615 ms total) 2024-09-10T20:27:44.798 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: 2024-09-10T20:27:44.798 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-10T20:27:44.798 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28595 ms total) 2024-09-10T20:27:44.798 INFO:tasks.workunit.client.0.smithi176.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-10T20:27:46.902 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-10T20:27:46.902 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-10T20:27:46.902 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-10T20:27:46.902 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5069 ms) 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (126 ms) 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20054 ms) 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: attempt 0 of 20 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (5368 ms) 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (30617 ms total) 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-10T20:27:46.903 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (30617 ms total) 2024-09-10T20:27:46.904 INFO:tasks.workunit.client.0.smithi176.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-10T20:27:50.304 INFO:tasks.workunit.client.0.smithi176.stdout: misc: Running main() from gmock_main.cc 2024-09-10T20:27:50.304 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-10T20:27:50.304 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [----------] Global test environment set-up. 2024-09-10T20:27:50.304 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-10T20:27:50.304 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-10T20:27:50.304 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.Version (1854 ms) 2024-09-10T20:27:50.304 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-10T20:27:50.304 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2011 ms) 2024-09-10T20:27:50.304 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.LongName (3206 ms) 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3020 ms) 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (2289 ms) 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (3038 ms) 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.Exec (2977 ms) 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (2977 ms) 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (3033 ms) 2024-09-10T20:27:50.305 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.BigObject (2969 ms) 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (2063 ms) 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (4484 ms) 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [----------] 12 tests from NeoRadosMisc (33921 ms total) 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [----------] Global test environment tear-down 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [==========] 12 tests from 1 test suite ran. (33921 ms total) 2024-09-10T20:27:50.306 INFO:tasks.workunit.client.0.smithi176.stdout: misc: [ PASSED ] 12 tests. 2024-09-10T20:27:55.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: Running main() from gmock_main.cc 2024-09-10T20:27:55.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-10T20:27:55.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [----------] Global test environment set-up. 2024-09-10T20:27:55.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-10T20:27:55.259 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-10T20:27:55.260 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ OK ] LibRadosPools.PoolList (3130 ms) 2024-09-10T20:27:55.260 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-10T20:27:55.260 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3177 ms) 2024-09-10T20:27:55.260 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-10T20:27:55.260 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3022 ms) 2024-09-10T20:27:55.261 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-10T20:27:55.261 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (2282 ms) 2024-09-10T20:27:55.261 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-10T20:27:55.261 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3068 ms) 2024-09-10T20:27:55.261 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-10T20:27:55.262 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (4950 ms) 2024-09-10T20:27:55.262 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-10T20:27:55.262 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (5040 ms) 2024-09-10T20:27:55.262 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-10T20:27:55.262 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5029 ms) 2024-09-10T20:27:55.262 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-10T20:27:55.263 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (9431 ms) 2024-09-10T20:27:55.263 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [----------] 9 tests from LibRadosPools (39130 ms total) 2024-09-10T20:27:55.263 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: 2024-09-10T20:27:55.263 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [----------] Global test environment tear-down 2024-09-10T20:27:55.263 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (39130 ms total) 2024-09-10T20:27:55.263 INFO:tasks.workunit.client.0.smithi176.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-10T20:27:57.781 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-10T20:27:57.781 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] Global test environment set-up. 2024-09-10T20:27:57.781 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-10T20:27:57.781 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-10T20:27:57.781 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-10T20:27:57.781 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: 2024-09-10T20:27:16.199+0000 7f5a2f229880 -1 failed for service _ceph-mon._tcp 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: 2024-09-10T20:27:16.199+0000 7f5a2f229880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (81 ms) 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5013 ms) 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5094 ms total) 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-10T20:27:57.782 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: started 0x561963040df0 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: started 0x561963c70680 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: started 2 aios 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: waiting 0x561963040df0 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: waiting 0x561963c70680 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: done. 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (6241 ms) 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (6241 ms total) 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-10T20:27:57.783 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMisc.Exec (302 ms) 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (18 ms) 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (6 ms) 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4655 ms) 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (23559 ms) 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28540 ms total) 2024-09-10T20:27:57.784 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: 2024-09-10T20:27:57.785 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [----------] Global test environment tear-down 2024-09-10T20:27:57.785 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (41629 ms total) 2024-09-10T20:27:57.785 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-10T20:27:58.774 INFO:tasks.workunit.client.0.smithi176.stdout: io: Running main() from gmock_main.cc 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [----------] Global test environment set-up. 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.Limits (2883 ms) 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3186 ms) 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.ReadOp (3032 ms) 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.SparseRead (2274 ms) 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.RoundTrip (3056 ms) 2024-09-10T20:27:58.775 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (2966 ms) 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3015 ms) 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (3010 ms) 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (2973 ms) 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.Trunc (3069 ms) 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.Remove (4469 ms) 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-10T20:27:58.776 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2504 ms) 2024-09-10T20:27:58.777 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-10T20:27:58.777 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.RmXattr (3455 ms) 2024-09-10T20:27:58.777 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-10T20:27:58.777 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2505 ms) 2024-09-10T20:27:58.777 INFO:tasks.workunit.client.0.smithi176.stdout: io: [----------] 14 tests from NeoRadosIo (42397 ms total) 2024-09-10T20:27:58.777 INFO:tasks.workunit.client.0.smithi176.stdout: io: 2024-09-10T20:27:58.777 INFO:tasks.workunit.client.0.smithi176.stdout: io: [----------] Global test environment tear-down 2024-09-10T20:27:58.777 INFO:tasks.workunit.client.0.smithi176.stdout: io: [==========] 14 tests from 1 test suite ran. (42398 ms total) 2024-09-10T20:27:58.777 INFO:tasks.workunit.client.0.smithi176.stdout: io: [ PASSED ] 14 tests. 2024-09-10T20:27:58.796 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: Running main() from gmock_main.cc 2024-09-10T20:27:58.796 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-10T20:27:58.796 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [----------] Global test environment set-up. 2024-09-10T20:27:58.796 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-10T20:27:58.796 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2817 ms) 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3196 ms) 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3036 ms) 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (2262 ms) 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (3056 ms) 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (2959 ms) 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3031 ms) 2024-09-10T20:27:58.797 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2979 ms) 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3003 ms) 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (3077 ms) 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (4399 ms) 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2529 ms) 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (3492 ms) 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (2505 ms) 2024-09-10T20:27:58.798 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (42342 ms total) 2024-09-10T20:27:58.799 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: 2024-09-10T20:27:58.799 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [----------] Global test environment tear-down 2024-09-10T20:27:58.799 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (42342 ms total) 2024-09-10T20:27:58.799 INFO:tasks.workunit.client.0.smithi176.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-10T20:28:14.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-10T20:28:14.247 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-10T20:28:14.248 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-10T20:28:14.248 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-10T20:28:14.248 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-10T20:28:14.248 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (2024 ms) 2024-09-10T20:28:14.248 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-10T20:28:14.249 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2180 ms) 2024-09-10T20:28:14.249 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-10T20:28:14.249 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (2028 ms) 2024-09-10T20:28:14.249 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-10T20:28:14.249 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2024 ms) 2024-09-10T20:28:14.249 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8256 ms total) 2024-09-10T20:28:14.250 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: 2024-09-10T20:28:14.250 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-10T20:28:14.250 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-10T20:28:14.250 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (4099 ms) 2024-09-10T20:28:14.250 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-10T20:28:14.250 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4001 ms) 2024-09-10T20:28:14.251 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-10T20:28:14.251 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (5109 ms) 2024-09-10T20:28:14.251 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (13209 ms total) 2024-09-10T20:28:14.251 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: 2024-09-10T20:28:14.251 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-10T20:28:14.252 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-10T20:28:14.252 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (4417 ms) 2024-09-10T20:28:14.252 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-10T20:28:14.252 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (2030 ms) 2024-09-10T20:28:14.252 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-10T20:28:14.252 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (1535 ms) 2024-09-10T20:28:14.253 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-10T20:28:14.253 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2450 ms) 2024-09-10T20:28:14.253 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (10432 ms total) 2024-09-10T20:28:14.253 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: 2024-09-10T20:28:14.253 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-10T20:28:14.253 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-10T20:28:14.254 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4093 ms) 2024-09-10T20:28:14.254 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-10T20:28:14.254 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3983 ms) 2024-09-10T20:28:14.254 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8076 ms total) 2024-09-10T20:28:14.254 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: 2024-09-10T20:28:14.254 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-10T20:28:14.254 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (58105 ms total) 2024-09-10T20:28:14.255 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-10T20:28:14.257 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:28:14.257 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28453 2024-09-10T20:28:14.257 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28453 2024-09-10T20:28:14.257 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:28:14.258 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28290 2024-09-10T20:28:14.258 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28290 2024-09-10T20:28:14.258 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:28:14.258 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28188 2024-09-10T20:28:14.258 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28188 2024-09-10T20:28:14.258 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:28:14.258 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28015 2024-09-10T20:28:14.259 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28015 2024-09-10T20:28:14.259 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:28:14.259 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28243 2024-09-10T20:28:14.259 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28243 2024-09-10T20:28:14.259 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:28:14.259 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28621 2024-09-10T20:28:14.260 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28621 2024-09-10T20:28:14.260 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:28:14.260 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28659 2024-09-10T20:28:14.260 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28659 2024-09-10T20:28:14.260 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:28:14.261 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28039 2024-09-10T20:28:14.261 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28039 2024-09-10T20:28:18.348 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T20:28:18.349 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T20:28:18.349 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T20:28:18.352 INFO:tasks.ceph.mon.c:Started 2024-09-10T20:28:18.352 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-09-10T20:28:18.353 INFO:tasks.ceph.mon.a:Sent signal 19 2024-09-10T20:28:18.353 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T20:28:29.416 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-10T20:28:29.416 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-10T20:28:29.416 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-10T20:28:29.416 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-10T20:28:29.416 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-10T20:28:29.416 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: seed 28060 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (1 ms) 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (314 ms) 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (43 ms) 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (10 ms) 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-10T20:28:29.417 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (10 ms) 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (7 ms) 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (3 ms) 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (13 ms) 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (4 ms) 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-10T20:28:29.418 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (14 ms) 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (25 ms) 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (4000 ms) 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (584 ms) 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: done waiting, doing copies 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: done waiting 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60879 ms) 2024-09-10T20:28:29.419 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (18 ms) 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (9 ms) 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4243 ms) 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-10T20:28:29.420 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-10T20:28:32.676 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 22 tests from LibRadosMis api_snapshots_pp: Running main() from gmock_main.cc 2024-09-10T20:28:32.676 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-10T20:28:32.676 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-10T20:28:32.676 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-10T20:28:32.676 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: seed 28135 2024-09-10T20:28:32.676 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (2025 ms) 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2181 ms) 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (2027 ms) 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2000 ms) 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (3302 ms) 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (11535 ms total) 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-10T20:28:32.677 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4120 ms) 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3936 ms) 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6966 ms) 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (4508 ms) 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (1594 ms) 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi176-28135-7 2024-09-10T20:28:32.678 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (18848 ms) 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (39972 ms total) 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (23 ms) 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4612 ms) 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4635 ms total) 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2501 ms) 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-10T20:28:32.679 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2008 ms) 2024-09-10T20:28:32.680 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-10T20:28:32.680 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (2272 ms) 2024-09-10T20:28:32.690 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_ snapshots: Running main() from gmock_main.cc 2024-09-10T20:28:32.690 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-10T20:28:32.690 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [----------] Global test environment set-up. 2024-09-10T20:28:32.690 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-10T20:28:32.690 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-10T20:28:32.690 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4868 ms) 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5204 ms) 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3275 ms) 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (4990 ms) 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7125 ms) 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25463 ms total) 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5880 ms) 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (6074 ms) 2024-09-10T20:28:32.691 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (8918 ms) 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (6009 ms) 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (4446 ms) 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi176-28837-11. 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: Waiting for snaps to purge. 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19482 ms) 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (50809 ms total) 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [----------] Global test environment tear-down 2024-09-10T20:28:32.692 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (76272 ms total) 2024-09-10T20:28:32.693 INFO:tasks.workunit.client.0.smithi176.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-10T20:28:33.353 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-09-10T20:28:33.354 INFO:tasks.ceph.mon.a:Sent signal 18 2024-09-10T20:28:33.354 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T20:28:33.354 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:28:33.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.474+0000 7fb12161f640 10 monclient: get_monmap_and_config 2024-09-10T20:28:33.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.474+0000 7fb12161f640 10 monclient: build_initial_monmap 2024-09-10T20:28:33.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.474+0000 7fb12161f640 10 monclient: monmap: 2024-09-10T20:28:33.482 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:33.482 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:33.482 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:33.480193+0000 2024-09-10T20:28:33.482 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:33.480193+0000 2024-09-10T20:28:33.482 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:33.482 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:33.483 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:33.483 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:33.483 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:33.483 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:33.483 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:33.483 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.474+0000 7fb12161f640 1 Processor -- start 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.474+0000 7fb12161f640 1 -- start start 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.474+0000 7fb12161f640 10 monclient: init 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.474+0000 7fb12161f640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 10 monclient: _add_conns ranks=[1,2,4,0,3] 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb11c0a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a4580 0x7fb11c13d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 10 monclient(hunting): picked mon.noname-b con 0x7fb11c0a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb11c0a49a0 0x7fb11c13fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:33.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 10 monclient(hunting): picked mon.noname-c con 0x7fb11c0a49a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb11c0a6a20 con 0x7fb11c0a41a0 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb11c0a2570 con 0x7fb11c0a49a0 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb11c070400 con 0x7fb11c0a4580 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb12161f640 10 monclient(hunting): authenticate will time out at 1073.620161s 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a4580 0x7fb11c13d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a4580 0x7fb11c13d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40824/0 (socket says 172.21.15.153:40824) 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1903444752 learned_addr learned my addr 172.21.15.153:0/1903444752 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient(hunting): get_auth_request con 0x7fb11c0a4580 auth_method 0 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:33.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient(hunting): handle_auth_done global_id 24116 payload 274 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1903444752 >> v2:172.21.15.176:3300/0 conn(0x7fb11c0a49a0 msgr2=0x7fb11c13fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 1 --2- 172.21.15.153:0/1903444752 >> v2:172.21.15.176:3300/0 conn(0x7fb11c0a49a0 0x7fb11c13fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1903444752 >> v1:172.21.15.176:6789/0 conn(0x7fb11c0a41a0 legacy=0x7fb11c0a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient: _finish_hunting 0 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 1 monclient: found mon.noname-b 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:28:33.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.478+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1903444752 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11c0a85a0 con 0x7fb11c0a4580 2024-09-10T20:28:33.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1903444752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a4580 msgr2=0x7fb11c13d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:33.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb11a7fc640 1 --2- 172.21.15.153:0/1903444752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a4580 0x7fb11c13d7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb110009870 tx=0x7fb11001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adf06e64d00057ce server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:33.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 1 -- 172.21.15.153:0/1903444752 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1100b0070 con 0x7fb11c0a4580 2024-09-10T20:28:33.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:33.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 10 monclient: dump: 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 10 monclient: _finish_auth 0 2024-09-10T20:28:33.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 1 -- 172.21.15.153:0/1903444752 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1100a97f0 con 0x7fb11c0a4580 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 5 monclient: authenticate success, global_id 24116 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 1 -- 172.21.15.153:0/1903444752 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1100a9bc0 con 0x7fb11c0a4580 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 4 set_mon_vals no callback set 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb119ffb640 10 monclient: dump: 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:33.490 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:33.491 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:33.491 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:33.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 10 monclient: shutdown 2024-09-10T20:28:33.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:33.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 -- 172.21.15.153:0/1903444752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a4580 msgr2=0x7fb11c13d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:33.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 --2- 172.21.15.153:0/1903444752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a4580 0x7fb11c13d7c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb110009870 tx=0x7fb11001c420 comp rx=0 tx=0).stop 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 -- 172.21.15.153:0/1903444752 shutdown_connections 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 --2- 172.21.15.153:0/1903444752 >> v2:172.21.15.176:3300/0 conn(0x7fb11c0a49a0 0x7fb11c13fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 --2- 172.21.15.153:0/1903444752 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a4580 0x7fb11c13d7c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 -- 172.21.15.153:0/1903444752 >> 172.21.15.153:0/1903444752 conn(0x7fb11c0ab7e0 msgr2=0x7fb11c0a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 -- 172.21.15.153:0/1903444752 shutdown_connections 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 -- 172.21.15.153:0/1903444752 wait complete. 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 10 monclient: build_initial_monmap 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 10 monclient: monmap: 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:33.496 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 librados: starting msgr at 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 librados: starting objecter 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 Processor -- start 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 -- start start 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 librados: setting wanted keys 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 1 librados: calling monclient init 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.482+0000 7fb12161f640 10 monclient: init 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:33.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb11c0a41a0 0x7fb11c1286f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient(hunting): picked mon.noname-a con 0x7fb11c0a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a49a0 0x7fb11c13f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient(hunting): picked mon.noname-c con 0x7fb11c0a49a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb11c13faa0 0x7fb11c1d04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient(hunting): picked mon.noname-b con 0x7fb11c13faa0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c142e00 con 0x7fb11c0a41a0 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb11c142c80 con 0x7fb11c13faa0 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb11c142f80 con 0x7fb11c0a49a0 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:33.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient(hunting): authenticate will time out at 1073.628627s 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a49a0 0x7fb11c13f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a49a0 0x7fb11c13f560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40830/0 (socket says 172.21.15.153:40830) 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1196071785 learned_addr learned my addr 172.21.15.153:0/1196071785 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient(hunting): get_auth_request con 0x7fb11c0a49a0 auth_method 0 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11affd640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb11c0a41a0 0x7fb11c1286f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11affd640 10 monclient(hunting): get_auth_request con 0x7fb11c0a41a0 auth_method 0 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:33.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11affd640 1 -- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb11c0a41a0 msgr2=0x7fb11c1286f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11affd640 1 -- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb11c0a41a0 msgr2=0x7fb11c1286f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11affd640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb11c0a41a0 0x7fb11c1286f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11affd640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb11c0a41a0 0x7fb11c1286f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11b7fe640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb11c13faa0 0x7fb11c1d04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11b7fe640 10 monclient(hunting): get_auth_request con 0x7fb11c13faa0 auth_method 0 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient(hunting): handle_auth_done global_id 24119 payload 824 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1196071785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb11c13faa0 msgr2=0x7fb11c1d04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb11c13faa0 0x7fb11c1d04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:33.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb11c0a41a0 msgr2=0x7fb11c1286f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb11c0a41a0 0x7fb11c1286f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient: _finish_hunting 0 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 monclient: found mon.noname-c 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1196071785 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb11c129ef0 con 0x7fb11c0a49a0 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 -- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a49a0 msgr2=0x7fb11c13f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb11a7fc640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a49a0 0x7fb11c13f560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb110009870 tx=0x7fb11001ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 client.?.objecter ms_handle_connect 0x7fb11c0a49a0 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 1 -- 172.21.15.153:0/1196071785 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1100b0040 con 0x7fb11c0a49a0 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:33.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 monclient: dump: 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 monclient: _finish_auth 0 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 monclient: _check_auth_tickets 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:33.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 1 -- 172.21.15.153:0/1196071785 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb110004030 con 0x7fb11c0a49a0 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 5 monclient: authenticate success, global_id 24119 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient: _renew_subs 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb11c0a21c0 con 0x7fb11c0a49a0 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 1 -- 172.21.15.153:0/1196071785 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1100043e0 con 0x7fb11c0a49a0 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1037fe640 10 monclient: dump: 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:33.503 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb1197fa640 4 set_mon_vals no callback set 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 client.24119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient: _renew_subs 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb11c129650 con 0x7fb11c0a49a0 2024-09-10T20:28:33.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.486+0000 7fb12161f640 1 librados: init done 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb1037fe640 1 -- 172.21.15.153:0/1196071785 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fb1100ab8c0 con 0x7fb11c0a49a0 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb12161f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb12161f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb12161f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb11c0a3720 con 0x7fb11c0a49a0 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb1037fe640 1 --2- 172.21.15.153:0/1196071785 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb10803cea0 0x7fb10803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb1037fe640 1 -- 172.21.15.153:0/1196071785 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(93..93 src has 1..93) ==== 7261+0+0 (secure 0 0 0) 0x7fb1100eade0 con 0x7fb11c0a49a0 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb1037fe640 10 client.24119.objecter ms_dispatch 0x7fb11c000ba0 osd_map(93..93 src has 1..93) 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb1037fe640 3 client.24119.objecter handle_osd_map got epochs [93,93] > 0 2024-09-10T20:28:33.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb1037fe640 3 client.24119.objecter handle_osd_map decoding full epoch 93 2024-09-10T20:28:33.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb1037fe640 20 client.24119.objecter dump_active .. 0 homeless 2024-09-10T20:28:33.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb11affd640 1 --2- 172.21.15.153:0/1196071785 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb10803cea0 0x7fb10803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:33.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.490+0000 7fb11affd640 10 monclient: get_auth_request con 0x7fb10803cea0 auth_method 0 2024-09-10T20:28:33.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.494+0000 7fb1037fe640 10 client.24119.objecter ms_handle_connect 0x7fb10803cea0 2024-09-10T20:28:33.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.494+0000 7fb11affd640 1 --2- 172.21.15.153:0/1196071785 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb10803cea0 0x7fb10803f360 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7fb10c007c40 tx=0x7fb10c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:33.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.494+0000 7fb1037fe640 1 -- 172.21.15.153:0/1196071785 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fb110016130 con 0x7fb11c0a49a0 2024-09-10T20:28:33.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.494+0000 7fb1037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:33.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.494+0000 7fb1037fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.726+0000 7fb12161f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:28:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.726+0000 7fb12161f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.726+0000 7fb12161f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:33.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.726+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb11c128c30 con 0x7fb11c0a49a0 2024-09-10T20:28:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb1037fe640 1 -- 172.21.15.153:0/1196071785 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fb1100e8e30 con 0x7fb11c0a49a0 2024-09-10T20:28:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb1037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:33.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb1037fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:33.736 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":14,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":9,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 10 librados: watch_flush enter 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 10 librados: watch_flush exit 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 20 client.24119.objecter shutdown clearing up homeless session... 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 10 client.24119.objecter successfully canceled tick 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb10803cea0 msgr2=0x7fb10803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 --2- 172.21.15.153:0/1196071785 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb10803cea0 0x7fb10803f360 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7fb10c007c40 tx=0x7fb10c009040 comp rx=0 tx=0).stop 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 10 monclient: shutdown 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a49a0 msgr2=0x7fb11c13f560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a49a0 0x7fb11c13f560 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb110009870 tx=0x7fb11001ac60 comp rx=0 tx=0).stop 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 shutdown_connections 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb11c13faa0 0x7fb11c1d04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 --2- 172.21.15.153:0/1196071785 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb10803cea0 0x7fb10803f360 unknown :-1 s=CLOSED pgs=3237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:33.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb11c0a49a0 0x7fb11c13f560 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 --2- 172.21.15.153:0/1196071785 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb11c0a41a0 0x7fb11c1286f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 >> 172.21.15.153:0/1196071785 conn(0x7fb11c0ab7e0 msgr2=0x7fb11c0a1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 shutdown_connections 2024-09-10T20:28:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 -- 172.21.15.153:0/1196071785 wait complete. 2024-09-10T20:28:33.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:33.730+0000 7fb12161f640 1 librados: shutdown 2024-09-10T20:28:36.756 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:28:36.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient: get_monmap_and_config 2024-09-10T20:28:36.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient: build_initial_monmap 2024-09-10T20:28:36.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient: monmap: 2024-09-10T20:28:36.885 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:36.883544+0000 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:36.883544+0000 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:36.886 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:36.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 1 Processor -- start 2024-09-10T20:28:36.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 1 -- start start 2024-09-10T20:28:36.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient: init 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient(hunting): picked mon.noname-c con 0x7f11b4131990 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f11b40a2c40 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient(hunting): picked mon.noname-a con 0x7f11b409f9c0 addr v2:172.21.15.153:3300/0 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11b409e2b0 con 0x7f11b40a2c40 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f11b41322b0 con 0x7f11b409f9c0 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f11b409d960 con 0x7f11b4131990 2024-09-10T20:28:36.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11ba27a640 10 monclient(hunting): authenticate will time out at 1077.023371s 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:42724/0 (socket says 172.21.15.153:42724) 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 learned_addr learned my addr 172.21.15.153:0/2185419863 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 10 monclient(hunting): get_auth_request con 0x7f11b409f9c0 auth_method 0 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 msgr2=0x7f11b40a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 msgr2=0x7f11b40a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:36.890 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b3fff640 10 monclient(hunting): get_auth_request con 0x7f11b4131990 auth_method 0 2024-09-10T20:28:36.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:36.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:36.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:36.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.878+0000 7f11b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:36.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.882+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 msgr2=0x7f11b4131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:28:36.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.882+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 msgr2=0x7f11b4131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:36.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.882+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:36.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.882+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:36.891 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:36.882+0000 7f11b37fe640 1 -- 172.21.15.153:0/2185419863 >> v1:172.21.15.176:6789/0 conn(0x7f11b40a2c40 legacy=0x7f11b40a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:37.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.082+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.082+0000 7f11b8817640 10 monclient(hunting): get_auth_request con 0x7f11b409f9c0 auth_method 0 2024-09-10T20:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.082+0000 7f11b8817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.082+0000 7f11b8817640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.082+0000 7f11b8817640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.082+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 msgr2=0x7f11b40a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.082+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 msgr2=0x7f11b40a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.082+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:37.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.082+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:28:37.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.086+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:37.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.086+0000 7f11b3fff640 10 monclient(hunting): get_auth_request con 0x7f11b4131990 auth_method 0 2024-09-10T20:28:37.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.086+0000 7f11b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:37.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.086+0000 7f11b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:37.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.086+0000 7f11b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:37.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.086+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 msgr2=0x7f11b4131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:28:37.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.086+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 msgr2=0x7f11b4131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:37.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.086+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:37.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.086+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:28:37.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.486+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:37.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.486+0000 7f11b8817640 10 monclient(hunting): get_auth_request con 0x7f11b409f9c0 auth_method 0 2024-09-10T20:28:37.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.486+0000 7f11b8817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:37.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.486+0000 7f11b8817640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:37.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.486+0000 7f11b8817640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:37.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.486+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 msgr2=0x7f11b40a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:28:37.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.486+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 msgr2=0x7f11b40a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:37.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.486+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:37.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.486+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:28:37.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.490+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:37.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.490+0000 7f11b3fff640 10 monclient(hunting): get_auth_request con 0x7f11b4131990 auth_method 0 2024-09-10T20:28:37.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.490+0000 7f11b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:37.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.490+0000 7f11b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:37.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.490+0000 7f11b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:37.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.490+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 msgr2=0x7f11b4131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:28:37.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.490+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 msgr2=0x7f11b4131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:37.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.490+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:37.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:37.490+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:28:38.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.290+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:38.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.290+0000 7f11b8817640 10 monclient(hunting): get_auth_request con 0x7f11b409f9c0 auth_method 0 2024-09-10T20:28:38.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.290+0000 7f11b8817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:38.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.290+0000 7f11b8817640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:38.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.290+0000 7f11b8817640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:38.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.290+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 msgr2=0x7f11b40a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:28:38.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.290+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 msgr2=0x7f11b40a1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:38.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.290+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:38.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.290+0000 7f11b8817640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:28:38.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.294+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:38.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.294+0000 7f11b3fff640 10 monclient(hunting): get_auth_request con 0x7f11b4131990 auth_method 0 2024-09-10T20:28:38.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.294+0000 7f11b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:38.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.294+0000 7f11b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:38.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.294+0000 7f11b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:38.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.294+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 msgr2=0x7f11b4131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:28:38.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.294+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 msgr2=0x7f11b4131d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:38.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.294+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:38.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.294+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:28:38.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1556778503 0 0) 0x7f11b409e2b0 con 0x7f11b40a2c40 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): _negotiate 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): my global_id is 19618 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f119c003530 con 0x7f11b40a2c40 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4287941213 0 0) 0x7f119c003530 con 0x7f11b40a2c40 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 msgr2=0x7f11b4131d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.176:3300/0 conn(0x7f11b4131990 0x7f11b4131d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 msgr2=0x7f11b40a1ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 --2- 172.21.15.153:0/2185419863 >> v2:172.21.15.153:3300/0 conn(0x7f11b409f9c0 0x7f11b40a1ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:28:38.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 monclient: found mon.noname-c-legacy 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.176:6789/0 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b40a2400 con 0x7f11b40a2c40 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient: _finish_auth 0 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3765011595 0 0) 0x7f11a4001a00 con 0x7f11b40a2c40 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient: dump: 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:38.637 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] but i'm connected to v1:172.21.15.176:6789/0, reconnecting 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 >> v1:172.21.15.176:6789/0 conn(0x7f11b40a2c40 legacy=0x7f11b40a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:28:38.638 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f119c003b30 0x7f119c003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): picked mon.b con 0x7f119c003b30 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119c004810 0x7f119c004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): picked mon.c con 0x7f119c004810 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119c005340 0x7f119c00b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): picked mon.a con 0x7f119c005340 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11b41322b0 con 0x7f119c005340 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f11b409d960 con 0x7f119c003b30 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f119c00c2e0 con 0x7f119c004810 2024-09-10T20:28:38.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b37fe640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119c004810 0x7f119c004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f119c003b30 0x7f119c003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient(hunting): get_auth_request con 0x7f119c003b30 auth_method 0 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b37fe640 10 monclient(hunting): get_auth_request con 0x7f119c004810 auth_method 0 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b37fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:38.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient(hunting): handle_auth_done global_id 19618 payload 293 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119c004810 msgr2=0x7f119c004c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119c004810 0x7f119c004c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119c005340 msgr2=0x7f119c00b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119c005340 0x7f119c00b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient: _finish_hunting 0 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 1 monclient: found mon.b 2024-09-10T20:28:38.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:28:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b8817640 1 -- 172.21.15.153:0/2185419863 reap_dead start 2024-09-10T20:28:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f119c003530 con 0x7f119c003b30 2024-09-10T20:28:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient: _finish_auth 0 2024-09-10T20:28:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 10 monclient: _check_auth_tickets 2024-09-10T20:28:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11ba27a640 5 monclient: authenticate success, global_id 19618 2024-09-10T20:28:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11ba27a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:28:38.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.626+0000 7f11b37fe640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f119c004810 0x7f119c004c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:28:38.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b3fff640 1 -- 172.21.15.153:0/2185419863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f119c003b30 msgr2=0x7f119c003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:38.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f119c003b30 0x7f119c003f50 secure :-1 s=READY pgs=5451 cs=0 l=1 rev1=1 crypto rx=0x7f11a8003880 tx=0x7f11a801eb70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d711f26d5a418940 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:38.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a80ad070 con 0x7f119c003b30 2024-09-10T20:28:38.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:38.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:28:38.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 10 monclient: dump: 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a80a7a70 con 0x7f119c003b30 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11ba27a640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:38.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 1 -- 172.21.15.153:0/2185419863 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a8004460 con 0x7f119c003b30 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11ba27a640 4 set_mon_vals no callback set 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.634+0000 7f11b2ffd640 10 monclient: dump: 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:38.645 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:38.646 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient: shutdown 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 -- 172.21.15.153:0/2185419863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f119c003b30 msgr2=0x7f119c003f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f119c003b30 0x7f119c003f50 secure :-1 s=READY pgs=5451 cs=0 l=1 rev1=1 crypto rx=0x7f11a8003880 tx=0x7f11a801eb70 comp rx=0 tx=0).stop 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 -- 172.21.15.153:0/2185419863 shutdown_connections 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 --2- 172.21.15.153:0/2185419863 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f119c003b30 0x7f119c003f50 unknown :-1 s=CLOSED pgs=5451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 -- 172.21.15.153:0/2185419863 >> 172.21.15.153:0/2185419863 conn(0x7f11b40a71b0 msgr2=0x7f11b40a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 -- 172.21.15.153:0/2185419863 shutdown_connections 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 -- 172.21.15.153:0/2185419863 wait complete. 2024-09-10T20:28:38.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient: build_initial_monmap 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient: monmap: 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 librados: starting msgr at 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 librados: starting objecter 2024-09-10T20:28:38.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 Processor -- start 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 -- start start 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 librados: setting wanted keys 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 librados: calling monclient init 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient: init 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119c003b30 0x7f11b41cdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient(hunting): picked mon.noname-a con 0x7f119c003b30 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11b41ce1f0 0x7f11b4122060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient(hunting): picked mon.noname-c con 0x7f11b41ce1f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b41225a0 0x7f11b4122a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient(hunting): picked mon.noname-b con 0x7f11b41225a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:38.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f11b4132100 con 0x7f119c003b30 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f11b4131f80 con 0x7f11b41225a0 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f11b40a2690 con 0x7f11b41ce1f0 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11ba27a640 10 monclient(hunting): authenticate will time out at 1078.783298s 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11b41ce1f0 0x7f11b4122060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11b41ce1f0 0x7f11b4122060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:40860/0 (socket says 172.21.15.153:40860) 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 1 -- 172.21.15.153:0/2533834086 learned_addr learned my addr 172.21.15.153:0/2533834086 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 10 monclient(hunting): get_auth_request con 0x7f11b41ce1f0 auth_method 0 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:38.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2533834086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119c003b30 0x7f11b41cdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b3fff640 10 monclient(hunting): get_auth_request con 0x7f119c003b30 auth_method 0 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b3fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b3fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b8817640 1 --2- 172.21.15.153:0/2533834086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b41225a0 0x7f11b4122a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b8817640 10 monclient(hunting): get_auth_request con 0x7f11b41225a0 auth_method 0 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b8817640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b8817640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b8817640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:38.653 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b3fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.638+0000 7f11b8817640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b8817640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b8817640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 10 monclient(hunting): handle_auth_done global_id 24122 payload 824 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 1 -- 172.21.15.153:0/2533834086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b41225a0 msgr2=0x7f11b4122a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 1 --2- 172.21.15.153:0/2533834086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b41225a0 0x7f11b4122a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 1 -- 172.21.15.153:0/2533834086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119c003b30 msgr2=0x7f11b41cdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 1 --2- 172.21.15.153:0/2533834086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119c003b30 0x7f11b41cdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 10 monclient: _finish_hunting 0 2024-09-10T20:28:38.654 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 1 monclient: found mon.noname-c 2024-09-10T20:28:38.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:38.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:28:38.655 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 1 -- 172.21.15.153:0/2533834086 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11b4123ad0 con 0x7f11b41ce1f0 2024-09-10T20:28:38.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 1 -- 172.21.15.153:0/2533834086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11b41ce1f0 msgr2=0x7f11b4122060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:38.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b37fe640 1 --2- 172.21.15.153:0/2533834086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11b41ce1f0 0x7f11b4122060 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f11a4011790 tx=0x7f11a4011b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:38.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 client.?.objecter ms_handle_connect 0x7f11b41ce1f0 2024-09-10T20:28:38.657 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 1 -- 172.21.15.153:0/2533834086 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a4019020 con 0x7f11b41ce1f0 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 monclient: dump: 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:38.658 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 monclient: _finish_auth 0 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 monclient: _check_auth_tickets 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 1 -- 172.21.15.153:0/2533834086 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11a400fa60 con 0x7f11b41ce1f0 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 1 -- 172.21.15.153:0/2533834086 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11a4010040 con 0x7f11b41ce1f0 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11ba27a640 5 monclient: authenticate success, global_id 24122 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11ba27a640 10 monclient: _renew_subs 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11ba27a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11b40a5260 con 0x7f11b41ce1f0 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b27fc640 4 set_mon_vals no callback set 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:38.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11b0ff9640 10 monclient: dump: 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11ba27a640 10 client.24122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:38.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11ba27a640 10 monclient: _renew_subs 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11ba27a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f11b4125590 con 0x7f11b41ce1f0 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.642+0000 7f11ba27a640 1 librados: init done 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11ba27a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11ba27a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11ba27a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11b40a29c0 con 0x7f11b41ce1f0 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11b0ff9640 1 -- 172.21.15.153:0/2533834086 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f11a401c020 con 0x7f11b41ce1f0 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11b0ff9640 1 --2- 172.21.15.153:0/2533834086 >> v2:172.21.15.176:6812/3003473398 conn(0x7f11a003ced0 0x7f11a003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11b0ff9640 1 -- 172.21.15.153:0/2533834086 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(93..93 src has 1..93) ==== 7261+0+0 (secure 0 0 0) 0x7f11a40529c0 con 0x7f11b41ce1f0 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11b0ff9640 10 client.24122.objecter ms_dispatch 0x7f11b4000ba0 osd_map(93..93 src has 1..93) 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11b0ff9640 3 client.24122.objecter handle_osd_map got epochs [93,93] > 0 2024-09-10T20:28:38.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11b0ff9640 3 client.24122.objecter handle_osd_map decoding full epoch 93 2024-09-10T20:28:38.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11b0ff9640 20 client.24122.objecter dump_active .. 0 homeless 2024-09-10T20:28:38.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2533834086 >> v2:172.21.15.176:6812/3003473398 conn(0x7f11a003ced0 0x7f11a003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:38.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.646+0000 7f11b3fff640 10 monclient: get_auth_request con 0x7f11a003ced0 auth_method 0 2024-09-10T20:28:38.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.650+0000 7f11b0ff9640 10 client.24122.objecter ms_handle_connect 0x7f11a003ced0 2024-09-10T20:28:38.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.650+0000 7f11b3fff640 1 --2- 172.21.15.153:0/2533834086 >> v2:172.21.15.176:6812/3003473398 conn(0x7f11a003ced0 0x7f11a003f390 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7f11a80af000 tx=0x7f11a8004740 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:38.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.650+0000 7f11b0ff9640 1 -- 172.21.15.153:0/2533834086 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f11a4023030 con 0x7f11b41ce1f0 2024-09-10T20:28:38.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.650+0000 7f11b0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:38.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.650+0000 7f11b0ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:38.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.906+0000 7f11ba27a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:28:38.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.906+0000 7f11ba27a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:38.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.906+0000 7f11ba27a640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:38.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.906+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f11b409e0a0 con 0x7f11b41ce1f0 2024-09-10T20:28:38.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.906+0000 7f11b0ff9640 1 -- 172.21.15.153:0/2533834086 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f11a4025300 con 0x7f11b41ce1f0 2024-09-10T20:28:38.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.906+0000 7f11b0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:38.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.906+0000 7f11b0ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:38.915 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":14,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":15,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:28:38.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 10 librados: watch_flush enter 2024-09-10T20:28:38.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 10 librados: watch_flush exit 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 20 client.24122.objecter shutdown clearing up homeless session... 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 10 client.24122.objecter successfully canceled tick 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 >> v2:172.21.15.176:6812/3003473398 conn(0x7f11a003ced0 msgr2=0x7f11a003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 --2- 172.21.15.153:0/2533834086 >> v2:172.21.15.176:6812/3003473398 conn(0x7f11a003ced0 0x7f11a003f390 secure :-1 s=READY pgs=3238 cs=0 l=1 rev1=1 crypto rx=0x7f11a80af000 tx=0x7f11a8004740 comp rx=0 tx=0).stop 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 10 monclient: shutdown 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11b41ce1f0 msgr2=0x7f11b4122060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 --2- 172.21.15.153:0/2533834086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11b41ce1f0 0x7f11b4122060 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f11a4011790 tx=0x7f11a4011b90 comp rx=0 tx=0).stop 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 shutdown_connections 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 --2- 172.21.15.153:0/2533834086 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f11b41225a0 0x7f11b4122a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 --2- 172.21.15.153:0/2533834086 >> v2:172.21.15.176:6812/3003473398 conn(0x7f11a003ced0 0x7f11a003f390 unknown :-1 s=CLOSED pgs=3238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 --2- 172.21.15.153:0/2533834086 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f11b41ce1f0 0x7f11b4122060 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 --2- 172.21.15.153:0/2533834086 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f119c003b30 0x7f11b41cdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:38.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 >> 172.21.15.153:0/2533834086 conn(0x7f11b40a71b0 msgr2=0x7f11b40a57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:38.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 shutdown_connections 2024-09-10T20:28:38.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 -- 172.21.15.153:0/2533834086 wait complete. 2024-09-10T20:28:38.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:38.910+0000 7f11ba27a640 1 librados: shutdown 2024-09-10T20:28:41.934 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:28:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient: get_monmap_and_config 2024-09-10T20:28:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient: build_initial_monmap 2024-09-10T20:28:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient: monmap: 2024-09-10T20:28:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:42.063301+0000 2024-09-10T20:28:42.065 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:42.063301+0000 2024-09-10T20:28:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:42.066 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:42.066 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 1 Processor -- start 2024-09-10T20:28:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 1 -- start start 2024-09-10T20:28:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient: init 2024-09-10T20:28:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient: _add_conns ranks=[1,4,3,0,2] 2024-09-10T20:28:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0c001364f0 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0c001360a0 0x7f0c0013b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:42.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient(hunting): picked mon.noname-c con 0x7f0c001360a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0c000a1990 0x7f0c0013d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient(hunting): picked mon.noname-a con 0x7f0c000a1990 addr v2:172.21.15.153:3300/0 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c000a27a0 con 0x7f0c001364f0 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f0c000a2550 con 0x7f0c000a1990 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0c000a2a80 con 0x7f0c001360a0 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c08365640 10 monclient(hunting): authenticate will time out at 1082.202954s 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0c000a1990 0x7f0c0013d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0c000a1990 0x7f0c0013d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54646/0 (socket says 172.21.15.153:54646) 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 1 -- 172.21.15.153:0/2529004386 learned_addr learned my addr 172.21.15.153:0/2529004386 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:42.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient(hunting): get_auth_request con 0x7f0c000a1990 auth_method 0 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c05900640 1 --2- 172.21.15.153:0/2529004386 >> v2:172.21.15.176:3300/0 conn(0x7f0c001360a0 0x7f0c0013b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient(hunting): handle_auth_done global_id 26217 payload 274 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 1 -- 172.21.15.153:0/2529004386 >> v2:172.21.15.176:3300/0 conn(0x7f0c001360a0 msgr2=0x7f0c0013b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 1 --2- 172.21.15.153:0/2529004386 >> v2:172.21.15.176:3300/0 conn(0x7f0c001360a0 0x7f0c0013b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 1 -- 172.21.15.153:0/2529004386 >> v1:172.21.15.176:6789/0 conn(0x7f0c001364f0 legacy=0x7f0c001368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient: _finish_hunting 0 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 1 monclient: found mon.noname-a 2024-09-10T20:28:42.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c06902640 1 -- 172.21.15.153:0/2529004386 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c000a2d60 con 0x7f0c000a1990 2024-09-10T20:28:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c05900640 10 monclient: get_auth_request con 0x7f0c001360a0 auth_method 0 2024-09-10T20:28:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.058+0000 7f0c05900640 1 --2- 172.21.15.153:0/2529004386 >> v2:172.21.15.176:3300/0 conn(0x7f0c001360a0 0x7f0c0013b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:28:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 1 -- 172.21.15.153:0/2529004386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c000a1990 msgr2=0x7f0c0013d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:42.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 1 --2- 172.21.15.153:0/2529004386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c000a1990 0x7f0c0013d970 secure :-1 s=READY pgs=8090 cs=0 l=1 rev1=1 crypto rx=0x7f0c000a13d0 tx=0x7f0bf801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b7f6070612d38f1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 1 -- 172.21.15.153:0/2529004386 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80a5750 con 0x7f0c000a1990 2024-09-10T20:28:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 10 monclient: dump: 2024-09-10T20:28:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:42.073 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 10 monclient: _finish_auth 0 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 10 monclient: _check_auth_tickets 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 1 -- 172.21.15.153:0/2529004386 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf80a58f0 con 0x7f0c000a1990 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 5 monclient: authenticate success, global_id 26217 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:42.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 4 set_mon_vals no callback set 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 1 -- 172.21.15.153:0/2529004386 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf80a5cc0 con 0x7f0c000a1990 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c050ff640 10 monclient: dump: 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:42.075 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient: shutdown 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 -- 172.21.15.153:0/2529004386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c000a1990 msgr2=0x7f0c0013d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 --2- 172.21.15.153:0/2529004386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c000a1990 0x7f0c0013d970 secure :-1 s=READY pgs=8090 cs=0 l=1 rev1=1 crypto rx=0x7f0c000a13d0 tx=0x7f0bf801c8b0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 -- 172.21.15.153:0/2529004386 shutdown_connections 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 --2- 172.21.15.153:0/2529004386 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c000a1990 0x7f0c0013d970 unknown :-1 s=CLOSED pgs=8090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 --2- 172.21.15.153:0/2529004386 >> v2:172.21.15.176:3300/0 conn(0x7f0c001360a0 0x7f0c0013b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 -- 172.21.15.153:0/2529004386 >> 172.21.15.153:0/2529004386 conn(0x7f0c000a71b0 msgr2=0x7f0c000a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 -- 172.21.15.153:0/2529004386 shutdown_connections 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 -- 172.21.15.153:0/2529004386 wait complete. 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient: build_initial_monmap 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient: monmap: 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:42.076 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 librados: starting msgr at 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 librados: starting objecter 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 Processor -- start 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 -- start start 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 librados: setting wanted keys 2024-09-10T20:28:42.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 librados: calling monclient init 2024-09-10T20:28:42.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient: init 2024-09-10T20:28:42.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:42.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:28:42.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c000a1990 0x7f0c001d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:42.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient(hunting): picked mon.noname-b con 0x7f0c000a1990 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:42.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c001360a0 0x7f0c001d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:42.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient(hunting): picked mon.noname-a con 0x7f0c001360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:42.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c001364f0 0x7f0c001d0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:42.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient(hunting): picked mon.noname-c con 0x7f0c001364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:42.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0c00140b40 con 0x7f0c001360a0 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0c001409c0 con 0x7f0c000a1990 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0c00140cc0 con 0x7f0c001364f0 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c08365640 10 monclient(hunting): authenticate will time out at 1082.210660s 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c001360a0 0x7f0c001d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c001360a0 0x7f0c001d0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54656/0 (socket says 172.21.15.153:54656) 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 1 -- 172.21.15.153:0/821520402 learned_addr learned my addr 172.21.15.153:0/821520402 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient(hunting): get_auth_request con 0x7f0c001360a0 auth_method 0 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:42.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c001364f0 0x7f0c001d0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 10 monclient(hunting): get_auth_request con 0x7f0c001364f0 auth_method 0 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06902640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06101640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c000a1990 0x7f0c001d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:42.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient(hunting): handle_auth_done global_id 26220 payload 824 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 1 -- 172.21.15.153:0/821520402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c001364f0 msgr2=0x7f0c001d0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c001364f0 0x7f0c001d0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 1 -- 172.21.15.153:0/821520402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c000a1990 msgr2=0x7f0c001d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c000a1990 0x7f0c001d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient: _finish_hunting 0 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 1 monclient: found mon.noname-a 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c05900640 1 -- 172.21.15.153:0/821520402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c00130fe0 con 0x7f0c001360a0 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06101640 10 monclient: get_auth_request con 0x7f0c000a1990 auth_method 0 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.066+0000 7f0c06101640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c000a1990 0x7f0c001d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c05900640 1 -- 172.21.15.153:0/821520402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c001360a0 msgr2=0x7f0c001d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:42.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c05900640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c001360a0 0x7f0c001d0890 secure :-1 s=READY pgs=8091 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000c7f0 tx=0x7f0bf000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 client.?.objecter ms_handle_connect 0x7f0c001360a0 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 1 -- 172.21.15.153:0/821520402 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0007c40 con 0x7f0c001360a0 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 monclient: dump: 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:42.086 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 monclient: _finish_auth 0 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 1 -- 172.21.15.153:0/821520402 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0bf0007de0 con 0x7f0c001360a0 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 5 monclient: authenticate success, global_id 26220 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 10 monclient: _renew_subs 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c000a67d0 con 0x7f0c001360a0 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 1 -- 172.21.15.153:0/821520402 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0011630 con 0x7f0c001360a0 2024-09-10T20:28:42.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 monclient: dump: 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:42.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 10 client.26220.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 10 monclient: _renew_subs 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c0009e0a0 con 0x7f0c001360a0 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 1 librados: init done 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf7fff640 4 set_mon_vals no callback set 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c001368d0 con 0x7f0c001360a0 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 1 -- 172.21.15.153:0/821520402 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f0bf00117d0 con 0x7f0c001360a0 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 1 --2- 172.21.15.153:0/821520402 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0be403cea0 0x7f0be403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 1 -- 172.21.15.153:0/821520402 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 6422+0+0 (secure 0 0 0) 0x7f0bf00505d0 con 0x7f0c001360a0 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 10 client.26220.objecter ms_dispatch 0x7f0c00000ba0 osd_map(97..97 src has 1..97) 2024-09-10T20:28:42.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 3 client.26220.objecter handle_osd_map got epochs [97,97] > 0 2024-09-10T20:28:42.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 3 client.26220.objecter handle_osd_map decoding full epoch 97 2024-09-10T20:28:42.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.070+0000 7f0bf6ffd640 20 client.26220.objecter dump_active .. 0 homeless 2024-09-10T20:28:42.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.074+0000 7f0c06101640 1 --2- 172.21.15.153:0/821520402 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0be403cea0 0x7f0be403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:42.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.074+0000 7f0c06101640 10 monclient: get_auth_request con 0x7f0be403cea0 auth_method 0 2024-09-10T20:28:42.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.074+0000 7f0c06101640 1 --2- 172.21.15.153:0/821520402 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0be403cea0 0x7f0be403f360 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc003e40 tx=0x7f0bfc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:42.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.074+0000 7f0bf6ffd640 10 client.26220.objecter ms_handle_connect 0x7f0be403cea0 2024-09-10T20:28:42.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.078+0000 7f0bf6ffd640 1 -- 172.21.15.153:0/821520402 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f0bf0015020 con 0x7f0c001360a0 2024-09-10T20:28:42.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.078+0000 7f0bf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:42.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.078+0000 7f0bf6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.310+0000 7f0c08365640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:28:42.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.310+0000 7f0c08365640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.310+0000 7f0c08365640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.310+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0c00132090 con 0x7f0c001360a0 2024-09-10T20:28:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.310+0000 7f0bf6ffd640 1 -- 172.21.15.153:0/821520402 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f0bf0010350 con 0x7f0c001360a0 2024-09-10T20:28:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.310+0000 7f0bf6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:42.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.310+0000 7f0bf6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:42.317 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":16,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:28:42.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 10 librados: watch_flush enter 2024-09-10T20:28:42.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 10 librados: watch_flush exit 2024-09-10T20:28:42.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 20 client.26220.objecter shutdown clearing up homeless session... 2024-09-10T20:28:42.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 10 client.26220.objecter successfully canceled tick 2024-09-10T20:28:42.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0be403cea0 msgr2=0x7f0be403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:42.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 --2- 172.21.15.153:0/821520402 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0be403cea0 0x7f0be403f360 secure :-1 s=READY pgs=3241 cs=0 l=1 rev1=1 crypto rx=0x7f0bfc003e40 tx=0x7f0bfc009040 comp rx=0 tx=0).stop 2024-09-10T20:28:42.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 10 monclient: shutdown 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c001360a0 msgr2=0x7f0c001d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c001360a0 0x7f0c001d0890 secure :-1 s=READY pgs=8091 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000c7f0 tx=0x7f0bf000ccb0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 shutdown_connections 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 --2- 172.21.15.153:0/821520402 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0be403cea0 0x7f0be403f360 unknown :-1 s=CLOSED pgs=3241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c001364f0 0x7f0c001d0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c001360a0 0x7f0c001d0890 unknown :-1 s=CLOSED pgs=8091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 --2- 172.21.15.153:0/821520402 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c000a1990 0x7f0c001d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 >> 172.21.15.153:0/821520402 conn(0x7f0c000a71b0 msgr2=0x7f0c000a5470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 shutdown_connections 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 -- 172.21.15.153:0/821520402 wait complete. 2024-09-10T20:28:42.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:42.314+0000 7f0c08365640 1 librados: shutdown 2024-09-10T20:28:45.337 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient: get_monmap_and_config 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient: build_initial_monmap 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient: monmap: 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:45.479186+0000 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:45.479186+0000 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.481 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:45.482 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:45.482 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:45.482 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:45.482 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:45.482 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 1 Processor -- start 2024-09-10T20:28:45.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 1 -- start start 2024-09-10T20:28:45.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient: init 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient: _add_conns ranks=[0,1,2,3,4] 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa8680a41a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa8680a4580 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient(hunting): picked mon.noname-b con 0x7fa8680a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8680a6a20 con 0x7fa8680a41a0 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8680a85a0 con 0x7fa8680a4580 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa8680a2570 con 0x7fa8680a49d0 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86f0e5640 10 monclient(hunting): authenticate will time out at 1085.619552s 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86d682640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86ce81640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fa8680a41a0 0x7fa8680a3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:47004/0 (socket says 172.21.15.153:47004) 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86ce81640 1 -- 172.21.15.153:0/1750933360 learned_addr learned my addr 172.21.15.153:0/1750933360 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86d682640 10 monclient(hunting): get_auth_request con 0x7fa8680a49d0 auth_method 0 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86d682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86d682640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86d682640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86d682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86d682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.474+0000 7fa86d682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.478+0000 7fa86d682640 10 monclient(hunting): handle_auth_done global_id 24155 payload 274 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.478+0000 7fa86d682640 1 -- 172.21.15.153:0/1750933360 >> v1:172.21.15.176:6789/0 conn(0x7fa8680a4580 legacy=0x7fa86813d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.478+0000 7fa86d682640 1 -- 172.21.15.153:0/1750933360 >> v1:172.21.15.153:6789/0 conn(0x7fa8680a41a0 legacy=0x7fa8680a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.478+0000 7fa86d682640 10 monclient: _finish_hunting 0 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.478+0000 7fa86d682640 1 monclient: found mon.noname-b 2024-09-10T20:28:45.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.478+0000 7fa86d682640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:45.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.478+0000 7fa86d682640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:28:45.486 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.478+0000 7fa86d682640 1 -- 172.21.15.153:0/1750933360 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa868143450 con 0x7fa8680a49d0 2024-09-10T20:28:45.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 -- 172.21.15.153:0/1750933360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 msgr2=0x7fa868140ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:45.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 --2- 172.21.15.153:0/1750933360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868140ef0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa860009dc0 tx=0x7fa86001c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=213c0d62d635eb5c server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 1 -- 172.21.15.153:0/1750933360 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600b2070 con 0x7fa8680a49d0 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 10 monclient: dump: 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.489 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 10 monclient: _finish_auth 0 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 10 monclient: _check_auth_tickets 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 1 -- 172.21.15.153:0/1750933360 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8600a5a30 con 0x7fa8680a49d0 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 5 monclient: authenticate success, global_id 24155 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 1 -- 172.21.15.153:0/1750933360 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600a5e00 con 0x7fa8680a49d0 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 4 set_mon_vals no callback set 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:45.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa8677fe640 10 monclient: dump: 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient: shutdown 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 -- 172.21.15.153:0/1750933360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 msgr2=0x7fa868140ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 --2- 172.21.15.153:0/1750933360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868140ef0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fa860009dc0 tx=0x7fa86001c850 comp rx=0 tx=0).stop 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 -- 172.21.15.153:0/1750933360 shutdown_connections 2024-09-10T20:28:45.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 --2- 172.21.15.153:0/1750933360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868140ef0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 -- 172.21.15.153:0/1750933360 >> 172.21.15.153:0/1750933360 conn(0x7fa8680ab7e0 msgr2=0x7fa8680a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:45.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 -- 172.21.15.153:0/1750933360 shutdown_connections 2024-09-10T20:28:45.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 -- 172.21.15.153:0/1750933360 wait complete. 2024-09-10T20:28:45.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient: build_initial_monmap 2024-09-10T20:28:45.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient: monmap: 2024-09-10T20:28:45.496 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:45.496 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:45.496 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:45.496 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:45.496 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:45.496 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 librados: starting msgr at 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 librados: starting objecter 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 Processor -- start 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 -- start start 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 librados: setting wanted keys 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 librados: calling monclient init 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient: init 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:45.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8680a41a0 0x7fa86813d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient(hunting): picked mon.noname-a con 0x7fa8680a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8680a4580 0x7fa868137ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient(hunting): picked mon.noname-b con 0x7fa8680a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868138010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient(hunting): picked mon.noname-c con 0x7fa8680a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa868144040 con 0x7fa8680a41a0 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa868070400 con 0x7fa8680a4580 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa8681441c0 con 0x7fa8680a49d0 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:45.498 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86f0e5640 10 monclient(hunting): authenticate will time out at 1085.626625s 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868138010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86ce81640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8680a41a0 0x7fa86813d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868138010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:36992/0 (socket says 172.21.15.153:36992) 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 -- 172.21.15.153:0/2916317459 learned_addr learned my addr 172.21.15.153:0/2916317459 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient(hunting): get_auth_request con 0x7fa8680a49d0 auth_method 0 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86ce81640 10 monclient(hunting): get_auth_request con 0x7fa8680a41a0 auth_method 0 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86ce81640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86ce81640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86ce81640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.499 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86ce81640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86ce81640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86ce81640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient(hunting): handle_auth_done global_id 24158 payload 824 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 -- 172.21.15.153:0/2916317459 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8680a4580 msgr2=0x7fa868137ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa867fff640 1 --2- 172.21.15.153:0/2916317459 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8680a4580 0x7fa868137ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 --2- 172.21.15.153:0/2916317459 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8680a4580 0x7fa868137ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 -- 172.21.15.153:0/2916317459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8680a41a0 msgr2=0x7fa86813d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 --2- 172.21.15.153:0/2916317459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8680a41a0 0x7fa86813d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient: _finish_hunting 0 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 monclient: found mon.noname-c 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:45.500 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86d682640 1 -- 172.21.15.153:0/2916317459 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8680a9770 con 0x7fa8680a49d0 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.482+0000 7fa86ce81640 1 --2- 172.21.15.153:0/2916317459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8680a41a0 0x7fa86813d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86d682640 1 -- 172.21.15.153:0/2916317459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 msgr2=0x7fa868138010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86d682640 1 --2- 172.21.15.153:0/2916317459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868138010 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa8600b0040 tx=0x7fa8600195f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 client.?.objecter ms_handle_connect 0x7fa8680a49d0 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 1 -- 172.21.15.153:0/2916317459 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600bd020 con 0x7fa8680a49d0 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 monclient: dump: 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.501 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 monclient: _finish_auth 0 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 monclient: _check_auth_tickets 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 1 -- 172.21.15.153:0/2916317459 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa8600adb80 con 0x7fa8680a49d0 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 1 -- 172.21.15.153:0/2916317459 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8600b2040 con 0x7fa8680a49d0 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 monclient: dump: 2024-09-10T20:28:45.502 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 5 monclient: authenticate success, global_id 24158 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 10 monclient: _renew_subs 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa866ffd640 4 set_mon_vals no callback set 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8680a9530 con 0x7fa8680a49d0 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 10 client.24158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:45.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 10 monclient: _renew_subs 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa86813eac0 con 0x7fa8680a49d0 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 1 librados: init done 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 1 -- 172.21.15.153:0/2916317459 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fa8600c0020 con 0x7fa8680a49d0 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8680aa510 con 0x7fa8680a49d0 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 1 --2- 172.21.15.153:0/2916317459 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa84803ced0 0x7fa84803f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 1 -- 172.21.15.153:0/2916317459 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(101..101 src has 1..101) ==== 6320+0+0 (secure 0 0 0) 0x7fa8600eafa0 con 0x7fa8680a49d0 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 10 client.24158.objecter ms_dispatch 0x7fa868000ba0 osd_map(101..101 src has 1..101) 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 3 client.24158.objecter handle_osd_map got epochs [101,101] > 0 2024-09-10T20:28:45.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 3 client.24158.objecter handle_osd_map decoding full epoch 101 2024-09-10T20:28:45.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa8657fa640 20 client.24158.objecter dump_active .. 0 homeless 2024-09-10T20:28:45.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86ce81640 1 --2- 172.21.15.153:0/2916317459 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa84803ced0 0x7fa84803f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.486+0000 7fa86ce81640 10 monclient: get_auth_request con 0x7fa84803ced0 auth_method 0 2024-09-10T20:28:45.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.490+0000 7fa8657fa640 10 client.24158.objecter ms_handle_connect 0x7fa84803ced0 2024-09-10T20:28:45.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.490+0000 7fa86ce81640 1 --2- 172.21.15.153:0/2916317459 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa84803ced0 0x7fa84803f390 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7fa850007ca0 tx=0x7fa850007bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:45.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.490+0000 7fa8657fa640 1 -- 172.21.15.153:0/2916317459 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fa8600c7030 con 0x7fa8680a49d0 2024-09-10T20:28:45.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.490+0000 7fa8657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:45.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.490+0000 7fa8657fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:45.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.726+0000 7fa86f0e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:28:45.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.726+0000 7fa86f0e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:45.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.726+0000 7fa86f0e5640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:45.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.726+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa86813e4f0 con 0x7fa8680a49d0 2024-09-10T20:28:45.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.726+0000 7fa8657fa640 1 -- 172.21.15.153:0/2916317459 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fa860016560 con 0x7fa8680a49d0 2024-09-10T20:28:45.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.726+0000 7fa8657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:45.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.726+0000 7fa8657fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:45.734 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:28:45.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.726+0000 7fa86f0e5640 10 librados: watch_flush enter 2024-09-10T20:28:45.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.726+0000 7fa86f0e5640 10 librados: watch_flush exit 2024-09-10T20:28:45.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 20 client.24158.objecter shutdown clearing up homeless session... 2024-09-10T20:28:45.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 10 client.24158.objecter successfully canceled tick 2024-09-10T20:28:45.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa84803ced0 msgr2=0x7fa84803f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 --2- 172.21.15.153:0/2916317459 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa84803ced0 0x7fa84803f390 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7fa850007ca0 tx=0x7fa850007bf0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 10 monclient: shutdown 2024-09-10T20:28:45.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:45.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 msgr2=0x7fa868138010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 --2- 172.21.15.153:0/2916317459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868138010 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fa8600b0040 tx=0x7fa8600195f0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 shutdown_connections 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 --2- 172.21.15.153:0/2916317459 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa84803ced0 0x7fa84803f390 unknown :-1 s=CLOSED pgs=3246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 --2- 172.21.15.153:0/2916317459 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa8680a49d0 0x7fa868138010 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 --2- 172.21.15.153:0/2916317459 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa8680a4580 0x7fa868137ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 --2- 172.21.15.153:0/2916317459 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa8680a41a0 0x7fa86813d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 >> 172.21.15.153:0/2916317459 conn(0x7fa8680ab7e0 msgr2=0x7fa86813aa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 shutdown_connections 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 -- 172.21.15.153:0/2916317459 wait complete. 2024-09-10T20:28:45.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.730+0000 7fa86f0e5640 1 librados: shutdown 2024-09-10T20:28:45.754 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T20:28:45.754 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T20:28:45.755 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:28:45.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient: get_monmap_and_config 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient: build_initial_monmap 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient: monmap: 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:45.874933+0000 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:45.874933+0000 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:45.877 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:45.878 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 1 Processor -- start 2024-09-10T20:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 1 -- start start 2024-09-10T20:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient: init 2024-09-10T20:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T20:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7f000a5ab0 0x7f7f000a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient(hunting): picked mon.noname-c con 0x7f7f000a5ab0 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:45.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7f000a63d0 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7f001390d0 0x7f7f00140c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient(hunting): picked mon.noname-a con 0x7f7f001390d0 addr v2:172.21.15.153:3300/0 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f00070780 con 0x7f7f000a63d0 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f7f000a3010 con 0x7f7f001390d0 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7f000709c0 con 0x7f7f000a5ab0 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f06755640 10 monclient(hunting): authenticate will time out at 1086.015032s 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7f001390d0 0x7f7f00140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7f001390d0 0x7f7f00140c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54662/0 (socket says 172.21.15.153:54662) 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 1 -- 172.21.15.153:0/3933948226 learned_addr learned my addr 172.21.15.153:0/3933948226 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient(hunting): get_auth_request con 0x7f7f001390d0 auth_method 0 2024-09-10T20:28:45.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7efffff640 1 --2- 172.21.15.153:0/3933948226 >> v2:172.21.15.176:3300/0 conn(0x7f7f000a5ab0 0x7f7f000a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7efffff640 10 monclient(hunting): get_auth_request con 0x7f7f000a5ab0 auth_method 0 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7efffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7efffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7efffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient(hunting): handle_auth_done global_id 26241 payload 274 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 1 -- 172.21.15.153:0/3933948226 >> v2:172.21.15.176:3300/0 conn(0x7f7f000a5ab0 msgr2=0x7f7f000a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:45.882 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 1 --2- 172.21.15.153:0/3933948226 >> v2:172.21.15.176:3300/0 conn(0x7f7f000a5ab0 0x7f7f000a5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 1 -- 172.21.15.153:0/3933948226 >> v1:172.21.15.176:6789/0 conn(0x7f7f000a63d0 legacy=0x7f7f001389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:45.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient: _finish_hunting 0 2024-09-10T20:28:45.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 1 monclient: found mon.noname-a 2024-09-10T20:28:45.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:45.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:45.883 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.870+0000 7f7f04cf2640 1 -- 172.21.15.153:0/3933948226 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f000a26e0 con 0x7f7f001390d0 2024-09-10T20:28:45.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.874+0000 7f7f04cf2640 1 -- 172.21.15.153:0/3933948226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f001390d0 msgr2=0x7f7f00140c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:45.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.874+0000 7f7f04cf2640 1 --2- 172.21.15.153:0/3933948226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f001390d0 0x7f7f00140c90 secure :-1 s=READY pgs=8097 cs=0 l=1 rev1=1 crypto rx=0x7f7f000a8560 tx=0x7f7ef801c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e5c62888d20f766 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:45.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 1 -- 172.21.15.153:0/3933948226 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80a5750 con 0x7f7f001390d0 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 10 monclient: dump: 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 10 monclient: _finish_auth 0 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 10 monclient: _check_auth_tickets 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 5 monclient: authenticate success, global_id 26241 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 1 -- 172.21.15.153:0/3933948226 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef80a58f0 con 0x7f7f001390d0 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 4 set_mon_vals no callback set 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 1 -- 172.21.15.153:0/3933948226 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef80a5cc0 con 0x7f7f001390d0 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efeffd640 10 monclient: dump: 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.886 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient: shutdown 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 -- 172.21.15.153:0/3933948226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f001390d0 msgr2=0x7f7f00140c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 --2- 172.21.15.153:0/3933948226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f001390d0 0x7f7f00140c90 secure :-1 s=READY pgs=8097 cs=0 l=1 rev1=1 crypto rx=0x7f7f000a8560 tx=0x7f7ef801c8b0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 -- 172.21.15.153:0/3933948226 shutdown_connections 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 --2- 172.21.15.153:0/3933948226 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f001390d0 0x7f7f00140c90 unknown :-1 s=CLOSED pgs=8097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 --2- 172.21.15.153:0/3933948226 >> v2:172.21.15.176:3300/0 conn(0x7f7f000a5ab0 0x7f7f000a5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.887 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 -- 172.21.15.153:0/3933948226 >> 172.21.15.153:0/3933948226 conn(0x7f7f000aba00 msgr2=0x7f7f000a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:45.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 -- 172.21.15.153:0/3933948226 shutdown_connections 2024-09-10T20:28:45.888 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 -- 172.21.15.153:0/3933948226 wait complete. 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient: build_initial_monmap 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient: monmap: 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:45.893 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 librados: starting msgr at 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 librados: starting objecter 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 Processor -- start 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 -- start start 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 librados: setting wanted keys 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 librados: calling monclient init 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient: init 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f000a5ab0 0x7f7f00137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient(hunting): picked mon.noname-a con 0x7f7f000a5ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f000a63d0 0x7f7f001377a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient(hunting): picked mon.noname-b con 0x7f7f000a63d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f001390d0 0x7f7f0013db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient(hunting): picked mon.noname-c con 0x7f7f001390d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f000a3960 con 0x7f7f000a5ab0 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7f000a3ae0 con 0x7f7f000a63d0 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7f001439f0 con 0x7f7f001390d0 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f06755640 10 monclient(hunting): authenticate will time out at 1086.022420s 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f000a5ab0 0x7f7f00137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f000a5ab0 0x7f7f00137260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54664/0 (socket says 172.21.15.153:54664) 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 1 -- 172.21.15.153:0/1536791344 learned_addr learned my addr 172.21.15.153:0/1536791344 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:45.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient(hunting): get_auth_request con 0x7f7f000a5ab0 auth_method 0 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f04cf2640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f001390d0 0x7f7f0013db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f04cf2640 10 monclient(hunting): get_auth_request con 0x7f7f001390d0 auth_method 0 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f04cf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f04cf2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f04cf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f04cf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f04cf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:45.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7f04cf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7eff7fe640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f000a63d0 0x7f7f001377a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7eff7fe640 10 monclient(hunting): get_auth_request con 0x7f7f000a63d0 auth_method 0 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7eff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7eff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7eff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient(hunting): handle_auth_done global_id 26244 payload 824 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f001390d0 msgr2=0x7f7f0013db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7f001390d0 0x7f7f0013db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f000a63d0 msgr2=0x7f7f001377a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7f000a63d0 0x7f7f001377a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient: _finish_hunting 0 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 1 monclient: found mon.noname-a 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:45.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.878+0000 7f7efffff640 1 -- 172.21.15.153:0/1536791344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f00122140 con 0x7f7f000a5ab0 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efffff640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f000a5ab0 msgr2=0x7f7f00137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efffff640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f000a5ab0 0x7f7f00137260 secure :-1 s=READY pgs=8098 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000d800 tx=0x7f7ef000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 client.?.objecter ms_handle_connect 0x7f7f000a5ab0 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef000b8d0 con 0x7f7f000a5ab0 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 monclient: dump: 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.898 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 monclient: _finish_auth 0 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 monclient: _check_auth_tickets 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ef0019070 con 0x7f7f000a5ab0 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 5 monclient: authenticate success, global_id 26244 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 10 monclient: _renew_subs 2024-09-10T20:28:45.899 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f000a9950 con 0x7f7f000a5ab0 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef000bc70 con 0x7f7f000a5ab0 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 monclient: dump: 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.900 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 10 client.26244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 10 monclient: _renew_subs 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efe7fc640 4 set_mon_vals no callback set 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f001401d0 con 0x7f7f000a5ab0 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 1 librados: init done 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e070 0x7f7f0013e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f000a20e0 con 0x7f7f0013e070 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e070 0x7f7f0013e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.901 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 10 monclient: get_auth_request con 0x7f7f0013e070 auth_method 0 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 10 monclient(hunting): handle_auth_done global_id 26247 payload 824 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7eff7fe640 1 -- 172.21.15.153:0/1536791344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f000a5e90 con 0x7f7f0013e070 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f7ef001e460 con 0x7f7f000a5ab0 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 1 --2- 172.21.15.153:0/1536791344 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7eec03cea0 0x7f7eec03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 6320+0+0 (secure 0 0 0) 0x7f7ef00501f0 con 0x7f7f000a5ab0 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 10 client.26244.objecter ms_dispatch 0x7f7f00000ba0 osd_map(101..101 src has 1..101) 2024-09-10T20:28:45.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 3 client.26244.objecter handle_osd_map got epochs [101,101] > 0 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 3 client.26244.objecter handle_osd_map decoding full epoch 101 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7efcff9640 20 client.26244.objecter dump_active .. 0 homeless 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f04cf2640 1 --2- 172.21.15.153:0/1536791344 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7eec03cea0 0x7f7eec03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.882+0000 7f7f04cf2640 10 monclient: get_auth_request con 0x7f7eec03cea0 auth_method 0 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 10 client.26244.objecter ms_handle_connect 0x7f7f0013e070 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 10 client.26244.objecter resend_mon_ops 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7eff7fe640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e070 msgr2=0x7f7f0013e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7eff7fe640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e070 0x7f7f0013e450 secure :-1 s=READY pgs=8099 cs=0 l=1 rev1=1 crypto rx=0x7f7f00140910 tx=0x7f7ef400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4007450 con 0x7f7f0013e070 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 10 monclient: dump: 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.903 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7ef4007450 con 0x7f7f0013e070 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e070 msgr2=0x7f7f0013e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e070 0x7f7f0013e450 secure :-1 s=READY pgs=8099 cs=0 l=1 rev1=1 crypto rx=0x7f7f00140910 tx=0x7f7ef400e960 comp rx=0 tx=0).stop 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e070 msgr2=0x7f7f0013e450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:45.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e070 0x7f7f0013e450 unknown :-1 s=CLOSED pgs=8099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7efcff9640 10 client.26244.objecter ms_handle_connect 0x7f7eec03cea0 2024-09-10T20:28:45.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.886+0000 7f7f04cf2640 1 --2- 172.21.15.153:0/1536791344 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7eec03cea0 0x7f7eec03f360 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f7f0006a960 tx=0x7f7ef80a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7f06755640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7f06755640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e990 0x7f7f00139c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7f06755640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7f000a6be0 con 0x7f7f0013e990 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e990 0x7f7f00139c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 10 monclient: get_auth_request con 0x7f7f0013e990 auth_method 0 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:45.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 10 monclient(hunting): handle_auth_done global_id 26250 payload 824 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.890+0000 7f7eff7fe640 1 -- 172.21.15.153:0/1536791344 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f0006a440 con 0x7f7f0013e990 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7eff7fe640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e990 msgr2=0x7f7f00139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7eff7fe640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e990 0x7f7f00139c50 secure :-1 s=READY pgs=8100 cs=0 l=1 rev1=1 crypto rx=0x7f7f00140910 tx=0x7f7ef4003770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 10 client.26244.objecter ms_handle_connect 0x7f7f0013e990 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 10 client.26244.objecter resend_mon_ops 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ef401ed80 con 0x7f7f0013e990 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 10 monclient: dump: 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.908 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4555 (secure 0 0 0) 0x7f7ef4028440 con 0x7f7f0013e990 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e990 msgr2=0x7f7f00139c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e990 0x7f7f00139c50 secure :-1 s=READY pgs=8100 cs=0 l=1 rev1=1 crypto rx=0x7f7f00140910 tx=0x7f7ef4003770 comp rx=0 tx=0).stop 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e990 msgr2=0x7f7f00139c50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:45.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7efcff9640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f0013e990 0x7f7f00139c50 secure :-1 s=CLOSED pgs=8100 cs=0 l=1 rev1=1 crypto rx=0x7f7f00140910 tx=0x7f7ef4003770 comp rx=0 tx=0).stop 2024-09-10T20:28:45.911 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 172583, 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-10T20:28:45.912 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:45.913 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:45.914 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:28:45.915 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:45.916 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.917 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:45.918 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.919 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.920 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:45.921 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:45.922 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:28:45.923 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:28:45.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 10 librados: watch_flush enter 2024-09-10T20:28:45.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 10 librados: watch_flush exit 2024-09-10T20:28:45.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 20 client.26244.objecter shutdown clearing up homeless session... 2024-09-10T20:28:45.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 10 client.26244.objecter successfully canceled tick 2024-09-10T20:28:45.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7eec03cea0 msgr2=0x7f7eec03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 1 --2- 172.21.15.153:0/1536791344 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7eec03cea0 0x7f7eec03f360 secure :-1 s=READY pgs=3247 cs=0 l=1 rev1=1 crypto rx=0x7f7f0006a960 tx=0x7f7ef80a7070 comp rx=0 tx=0).stop 2024-09-10T20:28:45.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 10 monclient: shutdown 2024-09-10T20:28:45.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:45.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f000a5ab0 msgr2=0x7f7f00137260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:45.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 1 --2- 172.21.15.153:0/1536791344 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7f000a5ab0 0x7f7f00137260 secure :-1 s=READY pgs=8098 cs=0 l=1 rev1=1 crypto rx=0x7f7ef000d800 tx=0x7f7ef000dcc0 comp rx=0 tx=0).stop 2024-09-10T20:28:45.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f04cf2640 1 -- 172.21.15.153:0/1536791344 reap_dead start 2024-09-10T20:28:45.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f04cf2640 1 -- 172.21.15.153:0/1536791344 reap_dead start 2024-09-10T20:28:45.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 shutdown_connections 2024-09-10T20:28:45.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 >> 172.21.15.153:0/1536791344 conn(0x7f7f000aba00 msgr2=0x7f7f0009cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:45.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 shutdown_connections 2024-09-10T20:28:45.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 1 -- 172.21.15.153:0/1536791344 wait complete. 2024-09-10T20:28:45.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:45.894+0000 7f7f06755640 1 librados: shutdown 2024-09-10T20:28:45.926 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:28:45.927 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T20:28:45.927 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T20:28:46.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.030+0000 7f517c85b640 10 monclient: get_monmap_and_config 2024-09-10T20:28:46.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.030+0000 7f517c85b640 10 monclient: build_initial_monmap 2024-09-10T20:28:46.039 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.030+0000 7f517c85b640 10 monclient: monmap: 2024-09-10T20:28:46.039 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:46.039 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.039 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:46.037259+0000 2024-09-10T20:28:46.040 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:46.037259+0000 2024-09-10T20:28:46.040 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:46.040 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.040 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:46.040 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:46.040 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:46.040 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:46.040 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:46.040 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 1 Processor -- start 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 1 -- start start 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 10 monclient: init 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 10 monclient: _add_conns ranks=[0,2,1,3,4] 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f51781364f0 addr v1:172.21.15.153:6789/0 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781360a0 0x7f517813b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 10 monclient(hunting): picked mon.noname-b con 0x7f51781360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f51780a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51780a27a0 con 0x7f51781364f0 2024-09-10T20:28:46.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51780a2a80 con 0x7f51780a1990 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f51780a2550 con 0x7f51781360a0 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517c85b640 10 monclient(hunting): authenticate will time out at 1086.176737s 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517659c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f51781364f0 0x7f51781368d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:47010/0 (socket says 172.21.15.153:47010) 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781360a0 0x7f517813b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f517659c640 1 -- 172.21.15.153:0/2822180754 learned_addr learned my addr 172.21.15.153:0/2822180754 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient(hunting): get_auth_request con 0x7f51781360a0 auth_method 0 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient(hunting): handle_auth_done global_id 24164 payload 274 2024-09-10T20:28:46.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 1 -- 172.21.15.153:0/2822180754 >> v1:172.21.15.176:6789/0 conn(0x7f51780a1990 legacy=0x7f517813d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:46.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 1 -- 172.21.15.153:0/2822180754 >> v1:172.21.15.153:6789/0 conn(0x7f51781364f0 legacy=0x7f51781368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:46.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient: _finish_hunting 0 2024-09-10T20:28:46.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 1 monclient: found mon.noname-b 2024-09-10T20:28:46.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:46.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.034+0000 7f516dd9b640 1 -- 172.21.15.153:0/2822180754 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51780a2d60 con 0x7f51781360a0 2024-09-10T20:28:46.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f516dd9b640 1 -- 172.21.15.153:0/2822180754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781360a0 msgr2=0x7f517813b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f516dd9b640 1 --2- 172.21.15.153:0/2822180754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781360a0 0x7f517813b420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f51600097f0 tx=0x7f516001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f530268b03a812d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 1 -- 172.21.15.153:0/2822180754 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51600b0070 con 0x7f51781360a0 2024-09-10T20:28:46.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:28:46.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 10 monclient: dump: 2024-09-10T20:28:46.045 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.045 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 10 monclient: _finish_auth 0 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 10 monclient: _check_auth_tickets 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 5 monclient: authenticate success, global_id 24164 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 1 -- 172.21.15.153:0/2822180754 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51600ab600 con 0x7f51781360a0 2024-09-10T20:28:46.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 1 -- 172.21.15.153:0/2822180754 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51600ab9d0 con 0x7f51781360a0 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 4 set_mon_vals no callback set 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f5175d9b640 10 monclient: dump: 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.047 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.048 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.048 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.048 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.048 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.048 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.048 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.048 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient: shutdown 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 -- 172.21.15.153:0/2822180754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781360a0 msgr2=0x7f517813b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 --2- 172.21.15.153:0/2822180754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781360a0 0x7f517813b420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f51600097f0 tx=0x7f516001c3a0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 -- 172.21.15.153:0/2822180754 shutdown_connections 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 --2- 172.21.15.153:0/2822180754 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781360a0 0x7f517813b420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 -- 172.21.15.153:0/2822180754 >> 172.21.15.153:0/2822180754 conn(0x7f51780a71b0 msgr2=0x7f51780a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 -- 172.21.15.153:0/2822180754 shutdown_connections 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 -- 172.21.15.153:0/2822180754 wait complete. 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient: build_initial_monmap 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient: monmap: 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:46.055 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 librados: starting msgr at 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 librados: starting objecter 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 Processor -- start 2024-09-10T20:28:46.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 -- start start 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 librados: setting wanted keys 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 librados: calling monclient init 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient: init 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51780a1990 0x7f517809e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient(hunting): picked mon.noname-c con 0x7f51780a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51781360a0 0x7f517809e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient(hunting): picked mon.noname-b con 0x7f51781360a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51781364f0 0x7f517809eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient(hunting): picked mon.noname-a con 0x7f51781364f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5178141d80 con 0x7f51781364f0 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5178141c00 con 0x7f51781360a0 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5178141f00 con 0x7f51780a1990 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517c85b640 10 monclient(hunting): authenticate will time out at 1086.183905s 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.038+0000 7f517659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51780a1990 0x7f517809e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f5176d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51781364f0 0x7f517809eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51780a1990 0x7f517809e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37024/0 (socket says 172.21.15.153:37024) 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 -- 172.21.15.153:0/4072424691 learned_addr learned my addr 172.21.15.153:0/4072424691 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f5176d9d640 10 monclient(hunting): get_auth_request con 0x7f51781364f0 auth_method 0 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f5176d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f5176d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f5176d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient(hunting): get_auth_request con 0x7f51780a1990 auth_method 0 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f5176d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f5176d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f5176d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51781360a0 0x7f517809e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): get_auth_request con 0x7f51781360a0 auth_method 0 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient(hunting): handle_auth_done global_id 24167 payload 824 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51781360a0 msgr2=0x7f517809e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f51781360a0 0x7f517809e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51781364f0 msgr2=0x7f517809eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51781364f0 0x7f517809eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient: _finish_hunting 0 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 monclient: found mon.noname-c 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 -- 172.21.15.153:0/4072424691 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f517809fb90 con 0x7f51780a1990 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f5176d9d640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f51781364f0 0x7f517809eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51780a1990 msgr2=0x7f517809e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517659c640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51780a1990 0x7f517809e0a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f516800c470 tx=0x7f516800c930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 client.?.objecter ms_handle_connect 0x7f51780a1990 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168007070 con 0x7f51780a1990 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 monclient: dump: 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.061 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 monclient: _finish_auth 0 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 5 monclient: authenticate success, global_id 24167 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 10 monclient: _renew_subs 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.062 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5168007210 con 0x7f51780a1990 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51780a6920 con 0x7f51780a1990 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5168007600 con 0x7f51780a1990 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 10 monclient: dump: 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.063 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517559a640 4 set_mon_vals no callback set 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 10 client.24167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 10 monclient: _renew_subs 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f517809f570 con 0x7f51780a1990 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 1 librados: init done 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781d0c10 0x7f51781d0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5178131220 con 0x7f51781d0c10 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781d0c10 0x7f51781d0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.064 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient: get_auth_request con 0x7f51781d0c10 auth_method 0 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.042+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f516801b400 con 0x7f51780a1990 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516dd9b640 10 monclient(hunting): handle_auth_done global_id 24170 payload 824 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516dd9b640 1 -- 172.21.15.153:0/4072424691 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51781368d0 con 0x7f51781d0c10 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 1 --2- 172.21.15.153:0/4072424691 >> v2:172.21.15.176:6812/3003473398 conn(0x7f515803ced0 0x7f515803f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(101..101 src has 1..101) ==== 6320+0+0 (secure 0 0 0) 0x7f516804fa60 con 0x7f51780a1990 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 10 client.24167.objecter ms_dispatch 0x7f5178000ba0 osd_map(101..101 src has 1..101) 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 3 client.24167.objecter handle_osd_map got epochs [101,101] > 0 2024-09-10T20:28:46.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 3 client.24167.objecter handle_osd_map decoding full epoch 101 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 20 client.24167.objecter dump_active .. 0 homeless 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f5176d9d640 1 --2- 172.21.15.153:0/4072424691 >> v2:172.21.15.176:6812/3003473398 conn(0x7f515803ced0 0x7f515803f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f5176d9d640 10 monclient: get_auth_request con 0x7f515803ced0 auth_method 0 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 10 client.24167.objecter ms_handle_connect 0x7f51781d0c10 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516dd9b640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781d0c10 msgr2=0x7f51781d0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 10 client.24167.objecter resend_mon_ops 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516dd9b640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781d0c10 0x7f51781d0ff0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f51600ad000 tx=0x7f5160019140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51600bb070 con 0x7f51781d0c10 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 10 monclient: dump: 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f51600b6640 con 0x7f51781d0c10 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:46.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:46.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781d0c10 msgr2=0x7f51781d0ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781d0c10 0x7f51781d0ff0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f51600ad000 tx=0x7f5160019140 comp rx=0 tx=0).stop 2024-09-10T20:28:46.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781d0c10 msgr2=0x7f51781d0ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:46.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51781d0c10 0x7f51781d0ff0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f5176d9d640 1 --2- 172.21.15.153:0/4072424691 >> v2:172.21.15.176:6812/3003473398 conn(0x7f515803ced0 0x7f515803f390 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f517000f220 tx=0x7f5170003d50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.046+0000 7f516f7fe640 10 client.24167.objecter ms_handle_connect 0x7f515803ced0 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f517c85b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f517c85b640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517802d450 0x7f517802d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f517c85b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5178070f40 con 0x7f517802d450 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517802d450 0x7f517802d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 10 monclient: get_auth_request con 0x7f517802d450 auth_method 0 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 10 monclient(hunting): handle_auth_done global_id 24173 payload 824 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 1 -- 172.21.15.153:0/4072424691 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f517806a660 con 0x7f517802d450 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517802d450 msgr2=0x7f517802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516dd9b640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517802d450 0x7f517802d860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f51600ae040 tx=0x7f5160009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516f7fe640 10 client.24167.objecter ms_handle_connect 0x7f517802d450 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516f7fe640 10 client.24167.objecter resend_mon_ops 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51600b0040 con 0x7f517802d450 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.054+0000 7f516f7fe640 10 monclient: dump: 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.071 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f51600b53e0 con 0x7f517802d450 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f516f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f516f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517802d450 msgr2=0x7f517802d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f516f7fe640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517802d450 0x7f517802d860 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f51600ae040 tx=0x7f5160009190 comp rx=0 tx=0).stop 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f516f7fe640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517802d450 msgr2=0x7f517802d860 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:46.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f516f7fe640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f517802d450 0x7f517802d860 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f51600ae040 tx=0x7f5160009190 comp rx=0 tx=0).stop 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 27623, 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.073 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:46.074 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:46.075 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:28:46.076 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:46.077 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:28:46.078 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.079 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:46.080 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:46.081 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:46.082 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:28:46.083 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 3 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.084 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:28:46.085 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:28:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 10 librados: watch_flush enter 2024-09-10T20:28:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 10 librados: watch_flush exit 2024-09-10T20:28:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 20 client.24167.objecter shutdown clearing up homeless session... 2024-09-10T20:28:46.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 10 client.24167.objecter successfully canceled tick 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 >> v2:172.21.15.176:6812/3003473398 conn(0x7f515803ced0 msgr2=0x7f515803f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 1 --2- 172.21.15.153:0/4072424691 >> v2:172.21.15.176:6812/3003473398 conn(0x7f515803ced0 0x7f515803f390 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f517000f220 tx=0x7f5170003d50 comp rx=0 tx=0).stop 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 10 monclient: shutdown 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51780a1990 msgr2=0x7f517809e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 1 --2- 172.21.15.153:0/4072424691 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f51780a1990 0x7f517809e0a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f516800c470 tx=0x7f516800c930 comp rx=0 tx=0).stop 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f5176d9d640 1 -- 172.21.15.153:0/4072424691 reap_dead start 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f5176d9d640 1 -- 172.21.15.153:0/4072424691 reap_dead start 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 shutdown_connections 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 >> 172.21.15.153:0/4072424691 conn(0x7f51780a71b0 msgr2=0x7f51780a56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 shutdown_connections 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 1 -- 172.21.15.153:0/4072424691 wait complete. 2024-09-10T20:28:46.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.058+0000 7f517c85b640 1 librados: shutdown 2024-09-10T20:28:46.088 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:28:46.088 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T20:28:46.088 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:28:46.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient: get_monmap_and_config 2024-09-10T20:28:46.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient: build_initial_monmap 2024-09-10T20:28:46.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient: monmap: 2024-09-10T20:28:46.209 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:46.209 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.209 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:46.206362+0000 2024-09-10T20:28:46.209 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:46.206362+0000 2024-09-10T20:28:46.209 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:46.210 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.210 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:46.210 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:46.210 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:46.210 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:46.210 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:46.211 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 1 Processor -- start 2024-09-10T20:28:46.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 1 -- start start 2024-09-10T20:28:46.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient: init 2024-09-10T20:28:46.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1b381364f0 0x7f1b381368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient(hunting): picked mon.noname-a con 0x7f1b381364f0 addr v2:172.21.15.153:3300/0 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1b381360a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1b380a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b3809dc30 con 0x7f1b381360a0 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b380a27a0 con 0x7f1b380a1990 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f1b380a2a80 con 0x7f1b381364f0 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3ed14640 10 monclient(hunting): authenticate will time out at 1086.346018s 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f1b381364f0 0x7f1b381368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b37fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f1b381360a0 0x7f1b3809ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:47026/0 (socket says 172.21.15.153:47026) 2024-09-10T20:28:46.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b37fff640 1 -- 172.21.15.153:0/4156336286 learned_addr learned my addr 172.21.15.153:0/4156336286 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient(hunting): get_auth_request con 0x7f1b381364f0 auth_method 0 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient(hunting): handle_auth_done global_id 26256 payload 274 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 1 -- 172.21.15.153:0/4156336286 >> v1:172.21.15.176:6789/0 conn(0x7f1b380a1990 legacy=0x7f1b3809d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 1 -- 172.21.15.153:0/4156336286 >> v1:172.21.15.153:6789/0 conn(0x7f1b381360a0 legacy=0x7f1b3809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient: _finish_hunting 0 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 1 monclient: found mon.noname-a 2024-09-10T20:28:46.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.202+0000 7f1b3cab0640 1 -- 172.21.15.153:0/4156336286 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b380a2d60 con 0x7f1b381364f0 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3cab0640 1 -- 172.21.15.153:0/4156336286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381364f0 msgr2=0x7f1b381368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3cab0640 1 --2- 172.21.15.153:0/4156336286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381364f0 0x7f1b381368d0 secure :-1 s=READY pgs=8103 cs=0 l=1 rev1=1 crypto rx=0x7f1b280097c0 tx=0x7f1b2801c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4c49a655bd95874 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 1 -- 172.21.15.153:0/4156336286 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b280b0070 con 0x7f1b381364f0 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 10 monclient: dump: 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.225 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 10 monclient: _finish_auth 0 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 10 monclient: _check_auth_tickets 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 1 -- 172.21.15.153:0/4156336286 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b280ab600 con 0x7f1b381364f0 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 5 monclient: authenticate success, global_id 26256 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:28:46.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 1 -- 172.21.15.153:0/4156336286 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b280ab9d0 con 0x7f1b381364f0 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 4 set_mon_vals no callback set 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b377fe640 10 monclient: dump: 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.227 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 10 monclient: shutdown 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 1 -- 172.21.15.153:0/4156336286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381364f0 msgr2=0x7f1b381368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 1 --2- 172.21.15.153:0/4156336286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381364f0 0x7f1b381368d0 secure :-1 s=READY pgs=8103 cs=0 l=1 rev1=1 crypto rx=0x7f1b280097c0 tx=0x7f1b2801c360 comp rx=0 tx=0).stop 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 1 -- 172.21.15.153:0/4156336286 shutdown_connections 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 1 --2- 172.21.15.153:0/4156336286 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381364f0 0x7f1b381368d0 unknown :-1 s=CLOSED pgs=8103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 1 -- 172.21.15.153:0/4156336286 >> 172.21.15.153:0/4156336286 conn(0x7f1b380a71b0 msgr2=0x7f1b380a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 1 -- 172.21.15.153:0/4156336286 shutdown_connections 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.206+0000 7f1b3ed14640 1 -- 172.21.15.153:0/4156336286 wait complete. 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient: build_initial_monmap 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient: monmap: 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:46.228 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 librados: starting msgr at 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 librados: starting objecter 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 Processor -- start 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 -- start start 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 librados: setting wanted keys 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 librados: calling monclient init 2024-09-10T20:28:46.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient: init 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b380a1990 0x7f1b3809e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient(hunting): picked mon.noname-c con 0x7f1b380a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381360a0 0x7f1b3809e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient(hunting): picked mon.noname-a con 0x7f1b381360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381364f0 0x7f1b3809eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient(hunting): picked mon.noname-b con 0x7f1b381364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f1b381420e0 con 0x7f1b381360a0 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1b38141f60 con 0x7f1b381364f0 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f1b38142260 con 0x7f1b380a1990 2024-09-10T20:28:46.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3ed14640 10 monclient(hunting): authenticate will time out at 1086.353391s 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381360a0 0x7f1b3809e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b380a1990 0x7f1b3809e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b380a1990 0x7f1b3809e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37042/0 (socket says 172.21.15.153:37042) 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 1 -- 172.21.15.153:0/2672974018 learned_addr learned my addr 172.21.15.153:0/2672974018 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient(hunting): get_auth_request con 0x7f1b380a1990 auth_method 0 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381360a0 0x7f1b3809e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54716/0 (socket says 172.21.15.153:54716) 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 10 monclient(hunting): get_auth_request con 0x7f1b381360a0 auth_method 0 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3d2b1640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381364f0 0x7f1b3809eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient(hunting): handle_auth_done global_id 24176 payload 824 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381364f0 msgr2=0x7f1b3809eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381364f0 0x7f1b3809eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381360a0 msgr2=0x7f1b3809e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381360a0 0x7f1b3809e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient: _finish_hunting 0 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 1 monclient: found mon.noname-c 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3cab0640 1 -- 172.21.15.153:0/2672974018 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3813d480 con 0x7f1b380a1990 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3d2b1640 10 monclient: get_auth_request con 0x7f1b381364f0 auth_method 0 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b3d2b1640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381364f0 0x7f1b3809eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.210+0000 7f1b37fff640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f1b381360a0 0x7f1b3809e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3cab0640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b380a1990 msgr2=0x7f1b3809e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3cab0640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b380a1990 0x7f1b3809e0a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b280ae040 tx=0x7f1b2801aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 client.?.objecter ms_handle_connect 0x7f1b380a1990 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b280b0050 con 0x7f1b380a1990 2024-09-10T20:28:46.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 monclient: dump: 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.234 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 monclient: _finish_auth 0 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 monclient: _check_auth_tickets 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b280bc070 con 0x7f1b380a1990 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b280b5c80 con 0x7f1b380a1990 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 5 monclient: authenticate success, global_id 24176 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 10 monclient: _renew_subs 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b380a6b80 con 0x7f1b380a1990 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.235 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b367fc640 4 set_mon_vals no callback set 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 monclient: dump: 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 10 client.24176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 10 monclient: _renew_subs 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b3809f590 con 0x7f1b380a1990 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 1 librados: init done 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d08b0 0x7f1b381d0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3813d040 con 0x7f1b381d08b0 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f1b280ab4c0 con 0x7f1b380a1990 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 1 --2- 172.21.15.153:0/2672974018 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1b2403cef0 0x7f1b2403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d08b0 0x7f1b381d0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 10 monclient: get_auth_request con 0x7f1b381d08b0 auth_method 0 2024-09-10T20:28:46.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(101..101 src has 1..101) ==== 6320+0+0 (secure 0 0 0) 0x7f1b280ea570 con 0x7f1b380a1990 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 10 client.24176.objecter ms_dispatch 0x7f1b38000ba0 osd_map(101..101 src has 1..101) 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 3 client.24176.objecter handle_osd_map got epochs [101,101] > 0 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 3 client.24176.objecter handle_osd_map decoding full epoch 101 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b357fa640 20 client.24176.objecter dump_active .. 0 homeless 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3d2b1640 1 --2- 172.21.15.153:0/2672974018 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1b2403cef0 0x7f1b2403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b3d2b1640 10 monclient: get_auth_request con 0x7f1b2403cef0 auth_method 0 2024-09-10T20:28:46.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 10 monclient(hunting): handle_auth_done global_id 19651 payload 824 2024-09-10T20:28:46.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.214+0000 7f1b37fff640 1 -- 172.21.15.153:0/2672974018 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b381368d0 con 0x7f1b381d08b0 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 10 client.24176.objecter ms_handle_connect 0x7f1b381d08b0 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 10 client.24176.objecter resend_mon_ops 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b37fff640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d08b0 msgr2=0x7f1b381d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b37fff640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d08b0 0x7f1b381d0c90 secure :-1 s=READY pgs=5468 cs=0 l=1 rev1=1 crypto rx=0x7f1b380a0290 tx=0x7f1b2c007700 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 10 client.24176.objecter ms_handle_connect 0x7f1b2403cef0 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b3d2b1640 1 --2- 172.21.15.153:0/2672974018 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1b2403cef0 0x7f1b2403f3b0 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f1b30007fd0 tx=0x7f1b30009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c00fce0 con 0x7f1b381d08b0 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 10 monclient: dump: 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.241 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1b2c00fce0 con 0x7f1b381d08b0 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d08b0 msgr2=0x7f1b381d0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d08b0 0x7f1b381d0c90 secure :-1 s=READY pgs=5468 cs=0 l=1 rev1=1 crypto rx=0x7f1b380a0290 tx=0x7f1b2c007700 comp rx=0 tx=0).stop 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d08b0 msgr2=0x7f1b381d0c90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.218+0000 7f1b357fa640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d08b0 0x7f1b381d0c90 secure :-1 s=CLOSED pgs=5468 cs=0 l=1 rev1=1 crypto rx=0x7f1b380a0290 tx=0x7f1b2c007700 comp rx=0 tx=0).stop 2024-09-10T20:28:46.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.222+0000 7f1b3ed14640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.222+0000 7f1b3ed14640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d11d0 0x7f1b38070840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.222+0000 7f1b3ed14640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.222+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3809fc20 con 0x7f1b381d11d0 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d11d0 0x7f1b38070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 10 monclient: get_auth_request con 0x7f1b381d11d0 auth_method 0 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 10 monclient(hunting): handle_auth_done global_id 19654 payload 824 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 1 -- 172.21.15.153:0/2672974018 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1b3806a640 con 0x7f1b381d11d0 2024-09-10T20:28:46.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d11d0 msgr2=0x7f1b38070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b37fff640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d11d0 0x7f1b38070840 secure :-1 s=READY pgs=5469 cs=0 l=1 rev1=1 crypto rx=0x7f1b380a0290 tx=0x7f1b2c004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b357fa640 10 client.24176.objecter ms_handle_connect 0x7f1b381d11d0 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b357fa640 10 client.24176.objecter resend_mon_ops 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c006030 con 0x7f1b381d11d0 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b357fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b357fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.226+0000 7f1b357fa640 10 monclient: dump: 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.244 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f1b2c006030 con 0x7f1b381d11d0 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b357fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b357fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d11d0 msgr2=0x7f1b38070840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b357fa640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d11d0 0x7f1b38070840 secure :-1 s=READY pgs=5469 cs=0 l=1 rev1=1 crypto rx=0x7f1b380a0290 tx=0x7f1b2c004950 comp rx=0 tx=0).stop 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b357fa640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d11d0 msgr2=0x7f1b38070840 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b357fa640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f1b381d11d0 0x7f1b38070840 unknown :-1 s=CLOSED pgs=5469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.245 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 172876, 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:28:46.246 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:28:46.247 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.248 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:28:46.249 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:28:46.250 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:28:46.251 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:46.252 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:28:46.253 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.254 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.255 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.256 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:46.257 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:28:46.258 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 10 librados: watch_flush enter 2024-09-10T20:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 10 librados: watch_flush exit 2024-09-10T20:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 20 client.24176.objecter shutdown clearing up homeless session... 2024-09-10T20:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 10 client.24176.objecter successfully canceled tick 2024-09-10T20:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1b2403cef0 msgr2=0x7f1b2403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 --2- 172.21.15.153:0/2672974018 >> v2:172.21.15.176:6812/3003473398 conn(0x7f1b2403cef0 0x7f1b2403f3b0 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f1b30007fd0 tx=0x7f1b30009040 comp rx=0 tx=0).stop 2024-09-10T20:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 10 monclient: shutdown 2024-09-10T20:28:46.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b380a1990 msgr2=0x7f1b3809e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b380a1990 0x7f1b3809e0a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1b280ae040 tx=0x7f1b2801aba0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3d2b1640 1 -- 172.21.15.153:0/2672974018 reap_dead start 2024-09-10T20:28:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 shutdown_connections 2024-09-10T20:28:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 --2- 172.21.15.153:0/2672974018 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f1b380a1990 0x7f1b3809e0a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 >> 172.21.15.153:0/2672974018 conn(0x7f1b380a71b0 msgr2=0x7f1b380a5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 shutdown_connections 2024-09-10T20:28:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 -- 172.21.15.153:0/2672974018 wait complete. 2024-09-10T20:28:46.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.230+0000 7f1b3ed14640 1 librados: shutdown 2024-09-10T20:28:46.261 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:28:46.261 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T20:28:46.261 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T20:28:46.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.366+0000 7f2a58396640 10 monclient: get_monmap_and_config 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.366+0000 7f2a58396640 10 monclient: build_initial_monmap 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.366+0000 7f2a58396640 10 monclient: monmap: 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:46.372879+0000 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:46.372879+0000 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:46.375 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 1 Processor -- start 2024-09-10T20:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 1 -- start start 2024-09-10T20:28:46.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient: init 2024-09-10T20:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T20:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2a5009a3b0 0x7f2a5009a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient(hunting): picked mon.noname-c con 0x7f2a5009a3b0 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2a500a1b00 addr v1:172.21.15.153:6789/0 2024-09-10T20:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a500a2620 0x7f2a5013e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient(hunting): picked mon.noname-b con 0x7f2a500a2620 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:46.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a5009dae0 con 0x7f2a500a1b00 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2a500a2f60 con 0x7f2a5009a3b0 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2a5009df60 con 0x7f2a500a2620 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a58396640 10 monclient(hunting): authenticate will time out at 1086.513529s 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a500a2620 0x7f2a5013e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a500a2620 0x7f2a5013e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37058/0 (socket says 172.21.15.153:37058) 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 1 -- 172.21.15.153:0/3035979955 learned_addr learned my addr 172.21.15.153:0/3035979955 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient(hunting): get_auth_request con 0x7f2a500a2620 auth_method 0 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56132640 1 --2- 172.21.15.153:0/3035979955 >> v2:172.21.15.176:3300/0 conn(0x7f2a5009a3b0 0x7f2a5009a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56132640 10 monclient(hunting): get_auth_request con 0x7f2a5009a3b0 auth_method 0 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56132640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56132640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56132640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56132640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56132640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56132640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient(hunting): handle_auth_done global_id 24179 payload 274 2024-09-10T20:28:46.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 1 -- 172.21.15.153:0/3035979955 >> v2:172.21.15.176:3300/0 conn(0x7f2a5009a3b0 msgr2=0x7f2a5009a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:46.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 1 --2- 172.21.15.153:0/3035979955 >> v2:172.21.15.176:3300/0 conn(0x7f2a5009a3b0 0x7f2a5009a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 1 -- 172.21.15.153:0/3035979955 >> v1:172.21.15.153:6789/0 conn(0x7f2a500a1b00 legacy=0x7f2a500a1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:46.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient: _finish_hunting 0 2024-09-10T20:28:46.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 1 monclient: found mon.noname-b 2024-09-10T20:28:46.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:46.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.370+0000 7f2a56933640 1 -- 172.21.15.153:0/3035979955 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5009eb30 con 0x7f2a500a2620 2024-09-10T20:28:46.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.374+0000 7f2a56933640 1 -- 172.21.15.153:0/3035979955 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a500a2620 msgr2=0x7f2a5013e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.374+0000 7f2a56933640 1 --2- 172.21.15.153:0/3035979955 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a500a2620 0x7f2a5013e4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009e50 tx=0x7f2a4801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fe16f88933ec18d server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 1 -- 172.21.15.153:0/3035979955 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a480a5530 con 0x7f2a500a2620 2024-09-10T20:28:46.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:28:46.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 10 monclient: dump: 2024-09-10T20:28:46.387 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.387 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.387 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.387 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.387 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.388 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.388 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.388 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.388 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.388 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 10 monclient: _finish_auth 0 2024-09-10T20:28:46.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 10 monclient: _check_auth_tickets 2024-09-10T20:28:46.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:46.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 1 -- 172.21.15.153:0/3035979955 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a480a56d0 con 0x7f2a500a2620 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 1 -- 172.21.15.153:0/3035979955 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a480a5aa0 con 0x7f2a500a2620 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 5 monclient: authenticate success, global_id 24179 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 4 set_mon_vals no callback set 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a55130640 10 monclient: dump: 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.389 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 10 monclient: shutdown 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 -- 172.21.15.153:0/3035979955 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a500a2620 msgr2=0x7f2a5013e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 --2- 172.21.15.153:0/3035979955 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a500a2620 0x7f2a5013e4c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f2a48009e50 tx=0x7f2a4801c8f0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 -- 172.21.15.153:0/3035979955 shutdown_connections 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 --2- 172.21.15.153:0/3035979955 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a500a2620 0x7f2a5013e4c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 --2- 172.21.15.153:0/3035979955 >> v2:172.21.15.176:3300/0 conn(0x7f2a5009a3b0 0x7f2a5009a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 -- 172.21.15.153:0/3035979955 >> 172.21.15.153:0/3035979955 conn(0x7f2a500a7100 msgr2=0x7f2a500a7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 -- 172.21.15.153:0/3035979955 shutdown_connections 2024-09-10T20:28:46.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 -- 172.21.15.153:0/3035979955 wait complete. 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 10 monclient: build_initial_monmap 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 10 monclient: monmap: 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 librados: starting msgr at 2024-09-10T20:28:46.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 librados: starting objecter 2024-09-10T20:28:46.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 Processor -- start 2024-09-10T20:28:46.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 -- start start 2024-09-10T20:28:46.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 librados: setting wanted keys 2024-09-10T20:28:46.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 1 librados: calling monclient init 2024-09-10T20:28:46.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.378+0000 7f2a58396640 10 monclient: init 2024-09-10T20:28:46.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:46.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:28:46.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a5009a3b0 0x7f2a501d09b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient(hunting): picked mon.noname-c con 0x7f2a5009a3b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:46.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a500a1b00 0x7f2a50099670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient(hunting): picked mon.noname-a con 0x7f2a500a1b00 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:46.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a500a2620 0x7f2a50099bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient(hunting): picked mon.noname-b con 0x7f2a500a2620 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2a50141220 con 0x7f2a500a1b00 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2a501413a0 con 0x7f2a500a2620 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2a50141520 con 0x7f2a5009a3b0 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a58396640 10 monclient(hunting): authenticate will time out at 1086.524240s 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a5009a3b0 0x7f2a501d09b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a5009a3b0 0x7f2a501d09b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37060/0 (socket says 172.21.15.153:37060) 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 1 -- 172.21.15.153:0/4264370644 learned_addr learned my addr 172.21.15.153:0/4264370644 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient(hunting): get_auth_request con 0x7f2a5009a3b0 auth_method 0 2024-09-10T20:28:46.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a55931640 1 --2- 172.21.15.153:0/4264370644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a500a1b00 0x7f2a50099670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a55931640 10 monclient(hunting): get_auth_request con 0x7f2a500a1b00 auth_method 0 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a55931640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a55931640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a55931640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a55931640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a55931640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a55931640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:46.396 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient(hunting): handle_auth_done global_id 24182 payload 824 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 1 -- 172.21.15.153:0/4264370644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a500a2620 msgr2=0x7f2a50099bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 1 --2- 172.21.15.153:0/4264370644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a500a2620 0x7f2a50099bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 1 -- 172.21.15.153:0/4264370644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a500a1b00 msgr2=0x7f2a50099670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 1 --2- 172.21.15.153:0/4264370644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a500a1b00 0x7f2a50099670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient: _finish_hunting 0 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 1 monclient: found mon.noname-c 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.382+0000 7f2a56132640 1 -- 172.21.15.153:0/4264370644 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a50074700 con 0x7f2a5009a3b0 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a56132640 1 -- 172.21.15.153:0/4264370644 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a5009a3b0 msgr2=0x7f2a501d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:46.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a56132640 1 --2- 172.21.15.153:0/4264370644 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a5009a3b0 0x7f2a501d09b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400e8a0 tx=0x7f2a4400ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 10 client.?.objecter ms_handle_connect 0x7f2a5009a3b0 2024-09-10T20:28:46.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:46.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 1 -- 172.21.15.153:0/4264370644 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44017070 con 0x7f2a5009a3b0 2024-09-10T20:28:46.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:46.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 10 monclient: dump: 2024-09-10T20:28:46.399 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.399 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.399 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 10 monclient: _finish_auth 0 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 1 -- 172.21.15.153:0/4264370644 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a4400cb50 con 0x7f2a5009a3b0 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a42ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:46.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.386+0000 7f2a5492f640 4 set_mon_vals no callback set 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a42ffd640 1 -- 172.21.15.153:0/4264370644 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44013470 con 0x7f2a5009a3b0 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a42ffd640 10 monclient: dump: 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:46.401 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a58396640 5 monclient: authenticate success, global_id 24182 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a58396640 10 monclient: _renew_subs 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a58396640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a5009e2f0 con 0x7f2a5009a3b0 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a58396640 10 client.24182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a58396640 10 monclient: _renew_subs 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a58396640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a500978a0 con 0x7f2a5009a3b0 2024-09-10T20:28:46.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.390+0000 7f2a58396640 1 librados: init done 2024-09-10T20:28:46.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.394+0000 7f2a42ffd640 1 -- 172.21.15.153:0/4264370644 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f2a4401a070 con 0x7f2a5009a3b0 2024-09-10T20:28:46.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.394+0000 7f2a42ffd640 1 --2- 172.21.15.153:0/4264370644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2a3403cef0 0x7f2a3403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:46.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.394+0000 7f2a42ffd640 1 -- 172.21.15.153:0/4264370644 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(102..102 src has 1..102) ==== 6418+0+0 (secure 0 0 0) 0x7f2a4404fcc0 con 0x7f2a5009a3b0 2024-09-10T20:28:46.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.394+0000 7f2a42ffd640 10 client.24182.objecter ms_dispatch 0x7f2a50000ba0 osd_map(102..102 src has 1..102) 2024-09-10T20:28:46.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.394+0000 7f2a42ffd640 3 client.24182.objecter handle_osd_map got epochs [102,102] > 0 2024-09-10T20:28:46.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.394+0000 7f2a42ffd640 3 client.24182.objecter handle_osd_map decoding full epoch 102 2024-09-10T20:28:46.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.394+0000 7f2a42ffd640 20 client.24182.objecter dump_active .. 0 homeless 2024-09-10T20:28:46.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.394+0000 7f2a55931640 1 --2- 172.21.15.153:0/4264370644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2a3403cef0 0x7f2a3403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:46.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.394+0000 7f2a55931640 10 monclient: get_auth_request con 0x7f2a3403cef0 auth_method 0 2024-09-10T20:28:46.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.398+0000 7f2a58396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.398+0000 7f2a58396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.398+0000 7f2a58396640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.398+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a5009a790 con 0x7f2a5009a3b0 2024-09-10T20:28:46.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.398+0000 7f2a55931640 1 --2- 172.21.15.153:0/4264370644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2a3403cef0 0x7f2a3403f3b0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c007fd0 tx=0x7f2a4c007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:46.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.398+0000 7f2a42ffd640 10 client.24182.objecter ms_handle_connect 0x7f2a3403cef0 2024-09-10T20:28:46.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.414+0000 7f2a42ffd640 1 -- 172.21.15.153:0/4264370644 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f2a50074700 con 0x7f2a5009a3b0 2024-09-10T20:28:46.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.414+0000 7f2a42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:46.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.414+0000 7f2a42ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.650+0000 7f2a58396640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T20:28:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.650+0000 7f2a58396640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T20:28:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.650+0000 7f2a58396640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:46.656 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.650+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f2a50095de0 con 0x7f2a5009a3b0 2024-09-10T20:28:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a42ffd640 1 -- 172.21.15.153:0/4264370644 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f2a44013dc0 con 0x7f2a5009a3b0 2024-09-10T20:28:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T20:28:46.660 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a42ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 10 librados: watch_flush enter 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 10 librados: watch_flush exit 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 20 client.24182.objecter shutdown clearing up homeless session... 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 10 client.24182.objecter successfully canceled tick 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2a3403cef0 msgr2=0x7f2a3403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 --2- 172.21.15.153:0/4264370644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2a3403cef0 0x7f2a3403f3b0 secure :-1 s=READY pgs=3250 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c007fd0 tx=0x7f2a4c007e40 comp rx=0 tx=0).stop 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 10 monclient: shutdown 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a5009a3b0 msgr2=0x7f2a501d09b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:46.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 --2- 172.21.15.153:0/4264370644 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a5009a3b0 0x7f2a501d09b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400e8a0 tx=0x7f2a4400ed60 comp rx=0 tx=0).stop 2024-09-10T20:28:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 shutdown_connections 2024-09-10T20:28:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 --2- 172.21.15.153:0/4264370644 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2a500a2620 0x7f2a50099bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 --2- 172.21.15.153:0/4264370644 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2a500a1b00 0x7f2a50099670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 --2- 172.21.15.153:0/4264370644 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2a3403cef0 0x7f2a3403f3b0 unknown :-1 s=CLOSED pgs=3250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 --2- 172.21.15.153:0/4264370644 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2a5009a3b0 0x7f2a501d09b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 >> 172.21.15.153:0/4264370644 conn(0x7f2a500a7100 msgr2=0x7f2a500a4f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 shutdown_connections 2024-09-10T20:28:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 -- 172.21.15.153:0/4264370644 wait complete. 2024-09-10T20:28:46.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:46.654+0000 7f2a58396640 1 librados: shutdown 2024-09-10T20:28:46.678 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T20:28:46.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.686+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.686+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=4217198404,config=3541143443,health=896152122,kv=3909119065,logm=1664400556}) 2024-09-10T20:28:46.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.686+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=8,config=1,health=12,kv=10,logm=69} crc {auth=3556286504,config=1228700967,health=427113707,kv=2406741671,logm=3667720016}) 2024-09-10T20:28:46.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.686+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.686+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=4217198404,config=3541143443,health=896152122,kv=3909119065,logm=1664400556}) 2024-09-10T20:28:46.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.686+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=7,config=1,health=11,kv=10,logm=71} crc {auth=1870035021,config=1228700967,health=77687778,kv=2837984843,logm=2293993191}) 2024-09-10T20:28:46.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499205919}) 2024-09-10T20:28:46.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3591293016}) 2024-09-10T20:28:46.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1499205919}) 2024-09-10T20:28:46.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=90181794}) 2024-09-10T20:28:46.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3164175543}) 2024-09-10T20:28:46.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3168758112}) 2024-09-10T20:28:46.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.698 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3164175543}) 2024-09-10T20:28:46.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.690+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1209015251}) 2024-09-10T20:28:46.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611054536}) 2024-09-10T20:28:46.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4042538161}) 2024-09-10T20:28:46.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3611054536}) 2024-09-10T20:28:46.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=773709455}) 2024-09-10T20:28:46.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234095728}) 2024-09-10T20:28:46.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2850470399}) 2024-09-10T20:28:46.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234095728}) 2024-09-10T20:28:46.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.694+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1447962165}) 2024-09-10T20:28:46.773 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.766+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.773 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.766+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407970557}) 2024-09-10T20:28:46.773 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.766+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3140356925}) 2024-09-10T20:28:46.773 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.766+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:46.773 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.766+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3407970557}) 2024-09-10T20:28:46.773 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:46.766+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3911373237}) 2024-09-10T20:28:47.133 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.126+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.126+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915687005}) 2024-09-10T20:28:47.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.126+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3435217821}) 2024-09-10T20:28:47.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.126+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.126+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=915687005}) 2024-09-10T20:28:47.134 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.126+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4056321544}) 2024-09-10T20:28:47.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.230+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.237 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.230+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288242693}) 2024-09-10T20:28:47.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.230+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=373519909}) 2024-09-10T20:28:47.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.230+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.230+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=288242693}) 2024-09-10T20:28:47.238 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.230+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3358219039}) 2024-09-10T20:28:47.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327053050}) 2024-09-10T20:28:47.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2048502715}) 2024-09-10T20:28:47.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3327053050}) 2024-09-10T20:28:47.279 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1805912941}) 2024-09-10T20:28:47.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2228924832}) 2024-09-10T20:28:47.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3355507687}) 2024-09-10T20:28:47.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2228924832}) 2024-09-10T20:28:47.281 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.274+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=148961913}) 2024-09-10T20:28:47.388 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.382+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.388 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.382+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838942321}) 2024-09-10T20:28:47.388 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.382+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4232641539}) 2024-09-10T20:28:47.388 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.382+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.388 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.382+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=838942321}) 2024-09-10T20:28:47.388 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.382+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3386241332}) 2024-09-10T20:28:47.393 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.386+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.393 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.386+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068590270}) 2024-09-10T20:28:47.393 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.386+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3777681091}) 2024-09-10T20:28:47.394 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.386+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.394 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.386+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4068590270}) 2024-09-10T20:28:47.394 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.386+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1169785042}) 2024-09-10T20:28:47.396 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.390+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.397 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.390+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=781411965}) 2024-09-10T20:28:47.397 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.390+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2251478691}) 2024-09-10T20:28:47.397 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.390+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.397 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.390+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=781411965}) 2024-09-10T20:28:47.397 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.390+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3884151924}) 2024-09-10T20:28:47.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.394+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.394+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=25} crc {logm=3372999110,mdsmap=3397941211,mgr=1914389234,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=2286438869,mon_config_key=3257012160,monmap=999369809,nvmeofgw=598337686,osd_metadata=2472831596,osd_pg_creating=121282160,osd_snap=3933986418,osdmap=4290979824}) 2024-09-10T20:28:47.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.394+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=8,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=14,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=13,osdmap=27} crc {logm=811821365,mdsmap=3397941211,mgr=3323764283,mgr_metadata=3232925821,mgrstat=2048668245,mon_config_key=2068929297,monmap=999369809,nvmeofgw=598337686,osd_metadata=753924036,osd_pg_creating=121282160,osd_snap=1162885991,osdmap=1454534362}) 2024-09-10T20:28:47.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.394+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.394+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=7,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=25} crc {logm=3372999110,mdsmap=3397941211,mgr=1914389234,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=2286438869,mon_config_key=3257012160,monmap=999369809,nvmeofgw=598337686,osd_metadata=2472831596,osd_pg_creating=121282160,osd_snap=3933986418,osdmap=4290979824}) 2024-09-10T20:28:47.400 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.394+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=6,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=14,osdmap=27} crc {logm=313295308,mdsmap=3397941211,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=2286438869,mon_config_key=1019571815,monmap=999369809,nvmeofgw=598337686,osd_metadata=3795740981,osd_pg_creating=121282160,osd_snap=1198519596,osdmap=672908827}) 2024-09-10T20:28:47.404 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.398+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.404 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.398+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=541975056}) 2024-09-10T20:28:47.404 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.398+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3215146263}) 2024-09-10T20:28:47.404 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.398+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.404 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.398+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=541975056}) 2024-09-10T20:28:47.404 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.398+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4283032433}) 2024-09-10T20:28:47.418 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.410+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.418 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.410+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=2499489226}) 2024-09-10T20:28:47.418 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.410+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=43} crc {osdmap=310973061}) 2024-09-10T20:28:47.418 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.410+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:28:47.418 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.410+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=44} crc {osdmap=2499489226}) 2024-09-10T20:28:47.418 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:28:47.410+0000 7f4f6b3a9640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=39} crc {osdmap=3225619025}) 2024-09-10T20:28:47.679 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T20:28:47.679 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:28:47.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient: get_monmap_and_config 2024-09-10T20:28:47.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient: build_initial_monmap 2024-09-10T20:28:47.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient: monmap: 2024-09-10T20:28:47.805 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:47.805 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:47.805 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:47.803476+0000 2024-09-10T20:28:47.806 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:47.803476+0000 2024-09-10T20:28:47.806 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:47.806 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:47.806 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:47.806 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:47.806 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:47.806 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:47.806 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:47.806 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:47.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 1 Processor -- start 2024-09-10T20:28:47.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 1 -- start start 2024-09-10T20:28:47.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient: init 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient: _add_conns ranks=[1,3,4,0,2] 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f16f81364f0 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f16f81360a0 0x7f16f813b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient(hunting): picked mon.noname-a con 0x7f16f81360a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f16f80a1990 0x7f16f813d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient(hunting): picked mon.noname-c con 0x7f16f80a1990 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16f80a27a0 con 0x7f16f81364f0 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f16f80a2550 con 0x7f16f81360a0 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:47.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f16f80a2a80 con 0x7f16f80a1990 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16fd02c640 10 monclient(hunting): authenticate will time out at 1087.943245s 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f16f81360a0 0x7f16f813b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f16f81360a0 0x7f16f813b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54730/0 (socket says 172.21.15.153:54730) 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 1 -- 172.21.15.153:0/3213598494 learned_addr learned my addr 172.21.15.153:0/3213598494 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 10 monclient(hunting): get_auth_request con 0x7f16f81360a0 auth_method 0 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f759e640 1 --2- 172.21.15.153:0/3213598494 >> v2:172.21.15.176:3300/0 conn(0x7f16f80a1990 0x7f16f813d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f759e640 10 monclient(hunting): get_auth_request con 0x7f16f80a1990 auth_method 0 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:47.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f759e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f759e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f759e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 10 monclient(hunting): handle_auth_done global_id 26274 payload 274 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 1 -- 172.21.15.153:0/3213598494 >> v2:172.21.15.176:3300/0 conn(0x7f16f80a1990 msgr2=0x7f16f813d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.798+0000 7f16f659c640 1 --2- 172.21.15.153:0/3213598494 >> v2:172.21.15.176:3300/0 conn(0x7f16f80a1990 0x7f16f813d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.802+0000 7f16f659c640 1 -- 172.21.15.153:0/3213598494 >> v1:172.21.15.176:6789/0 conn(0x7f16f81364f0 legacy=0x7f16f81368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.802+0000 7f16f659c640 10 monclient: _finish_hunting 0 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.802+0000 7f16f659c640 1 monclient: found mon.noname-a 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.802+0000 7f16f659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:47.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.802+0000 7f16f659c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:47.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.802+0000 7f16f659c640 1 -- 172.21.15.153:0/3213598494 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f80a2d60 con 0x7f16f81360a0 2024-09-10T20:28:47.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 1 -- 172.21.15.153:0/3213598494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f81360a0 msgr2=0x7f16f813b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:47.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 1 --2- 172.21.15.153:0/3213598494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f81360a0 0x7f16f813b420 secure :-1 s=READY pgs=8109 cs=0 l=1 rev1=1 crypto rx=0x7f16f81370d0 tx=0x7f16e801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3dd34074f93f86b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:47.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 1 -- 172.21.15.153:0/3213598494 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e80b1070 con 0x7f16f81360a0 2024-09-10T20:28:47.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 10 monclient: dump: 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 10 monclient: _finish_auth 0 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 10 monclient: _check_auth_tickets 2024-09-10T20:28:47.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 5 monclient: authenticate success, global_id 26274 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 1 -- 172.21.15.153:0/3213598494 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16e80ac620 con 0x7f16f81360a0 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 4 set_mon_vals no callback set 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 1 -- 172.21.15.153:0/3213598494 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16e80ac9f0 con 0x7f16f81360a0 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f5d9b640 10 monclient: dump: 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:47.815 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient: shutdown 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 -- 172.21.15.153:0/3213598494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f81360a0 msgr2=0x7f16f813b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 --2- 172.21.15.153:0/3213598494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f81360a0 0x7f16f813b420 secure :-1 s=READY pgs=8109 cs=0 l=1 rev1=1 crypto rx=0x7f16f81370d0 tx=0x7f16e801c3e0 comp rx=0 tx=0).stop 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 -- 172.21.15.153:0/3213598494 shutdown_connections 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 --2- 172.21.15.153:0/3213598494 >> v2:172.21.15.176:3300/0 conn(0x7f16f80a1990 0x7f16f813d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 --2- 172.21.15.153:0/3213598494 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f81360a0 0x7f16f813b420 unknown :-1 s=CLOSED pgs=8109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:47.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 -- 172.21.15.153:0/3213598494 >> 172.21.15.153:0/3213598494 conn(0x7f16f80a71b0 msgr2=0x7f16f80a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 -- 172.21.15.153:0/3213598494 shutdown_connections 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 -- 172.21.15.153:0/3213598494 wait complete. 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient: build_initial_monmap 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient: monmap: 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:47.817 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:47.818 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:47.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 librados: starting msgr at 2024-09-10T20:28:47.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 librados: starting objecter 2024-09-10T20:28:47.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 Processor -- start 2024-09-10T20:28:47.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 -- start start 2024-09-10T20:28:47.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 librados: setting wanted keys 2024-09-10T20:28:47.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 librados: calling monclient init 2024-09-10T20:28:47.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient: init 2024-09-10T20:28:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:28:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f80a1990 0x7f16f8073cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient(hunting): picked mon.noname-a con 0x7f16f80a1990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16f81364f0 0x7f16f813c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient(hunting): picked mon.noname-c con 0x7f16f81364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16f813cbf0 0x7f16f81ce350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient(hunting): picked mon.noname-b con 0x7f16f813cbf0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:47.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f16f8140ae0 con 0x7f16f80a1990 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f16f8140960 con 0x7f16f813cbf0 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f16f8140c60 con 0x7f16f81364f0 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16fd02c640 10 monclient(hunting): authenticate will time out at 1087.950599s 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f80a1990 0x7f16f8073cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f80a1990 0x7f16f8073cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54744/0 (socket says 172.21.15.153:54744) 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 1 -- 172.21.15.153:0/4098022291 learned_addr learned my addr 172.21.15.153:0/4098022291 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient(hunting): get_auth_request con 0x7f16f80a1990 auth_method 0 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:47.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 1 --2- 172.21.15.153:0/4098022291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16f81364f0 0x7f16f813c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 10 monclient(hunting): get_auth_request con 0x7f16f81364f0 auth_method 0 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f759e640 1 --2- 172.21.15.153:0/4098022291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16f813cbf0 0x7f16f81ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f759e640 10 monclient(hunting): get_auth_request con 0x7f16f813cbf0 auth_method 0 2024-09-10T20:28:47.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient(hunting): handle_auth_done global_id 26277 payload 824 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 1 -- 172.21.15.153:0/4098022291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16f81364f0 msgr2=0x7f16f813c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 1 --2- 172.21.15.153:0/4098022291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16f81364f0 0x7f16f813c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 1 -- 172.21.15.153:0/4098022291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16f813cbf0 msgr2=0x7f16f81ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 1 --2- 172.21.15.153:0/4098022291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16f813cbf0 0x7f16f81ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient: _finish_hunting 0 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 1 monclient: found mon.noname-a 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.806+0000 7f16f6d9d640 1 -- 172.21.15.153:0/4098022291 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16f809e0a0 con 0x7f16f80a1990 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16f6d9d640 1 -- 172.21.15.153:0/4098022291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f80a1990 msgr2=0x7f16f8073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:47.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16f6d9d640 1 --2- 172.21.15.153:0/4098022291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f80a1990 0x7f16f8073cb0 secure :-1 s=READY pgs=8110 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00c450 tx=0x7f16ec00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 client.?.objecter ms_handle_connect 0x7f16f80a1990 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 1 -- 172.21.15.153:0/4098022291 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec0119a0 con 0x7f16f80a1990 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 monclient: dump: 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:47.826 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 monclient: _finish_auth 0 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 1 -- 172.21.15.153:0/4098022291 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16ec017070 con 0x7f16f80a1990 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 1 -- 172.21.15.153:0/4098022291 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16ec011da0 con 0x7f16f80a1990 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 5 monclient: authenticate success, global_id 26277 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 monclient: dump: 2024-09-10T20:28:47.827 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 10 monclient: _renew_subs 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16f559a640 4 set_mon_vals no callback set 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16f8074670 con 0x7f16f80a1990 2024-09-10T20:28:47.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 10 client.26277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 10 monclient: _renew_subs 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16f8138b20 con 0x7f16f80a1990 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 1 librados: init done 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16f8139e10 con 0x7f16f80a1990 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 1 -- 172.21.15.153:0/4098022291 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f16ec01c460 con 0x7f16f80a1990 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 1 --2- 172.21.15.153:0/4098022291 >> v2:172.21.15.176:6812/3003473398 conn(0x7f16e403cea0 0x7f16e403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:47.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 1 -- 172.21.15.153:0/4098022291 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6016+0+0 (secure 0 0 0) 0x7f16ec04fac0 con 0x7f16f80a1990 2024-09-10T20:28:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 10 client.26277.objecter ms_dispatch 0x7f16f8000ba0 osd_map(103..103 src has 1..103) 2024-09-10T20:28:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 3 client.26277.objecter handle_osd_map got epochs [103,103] > 0 2024-09-10T20:28:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 3 client.26277.objecter handle_osd_map decoding full epoch 103 2024-09-10T20:28:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16df7fe640 20 client.26277.objecter dump_active .. 0 homeless 2024-09-10T20:28:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16f659c640 1 --2- 172.21.15.153:0/4098022291 >> v2:172.21.15.176:6812/3003473398 conn(0x7f16e403cea0 0x7f16e403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.810+0000 7f16f659c640 10 monclient: get_auth_request con 0x7f16e403cea0 auth_method 0 2024-09-10T20:28:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.814+0000 7f16df7fe640 10 client.26277.objecter ms_handle_connect 0x7f16e403cea0 2024-09-10T20:28:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.814+0000 7f16f659c640 1 --2- 172.21.15.153:0/4098022291 >> v2:172.21.15.176:6812/3003473398 conn(0x7f16e403cea0 0x7f16e403f360 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7f16f80a12f0 tx=0x7f16e80a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:47.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.818+0000 7f16df7fe640 1 -- 172.21.15.153:0/4098022291 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f16f8139e10 con 0x7f16f80a1990 2024-09-10T20:28:47.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.818+0000 7f16df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:47.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:47.818+0000 7f16df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16fd02c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:28:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16fd02c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16fd02c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f16f81368d0 con 0x7f16f80a1990 2024-09-10T20:28:48.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16df7fe640 1 -- 172.21.15.153:0/4098022291 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f16ec01c9e0 con 0x7f16f80a1990 2024-09-10T20:28:48.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:48.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:48.057 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:28:48.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16fd02c640 10 librados: watch_flush enter 2024-09-10T20:28:48.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16fd02c640 10 librados: watch_flush exit 2024-09-10T20:28:48.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16fd02c640 20 client.26277.objecter shutdown clearing up homeless session... 2024-09-10T20:28:48.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.050+0000 7f16fd02c640 10 client.26277.objecter successfully canceled tick 2024-09-10T20:28:48.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 >> v2:172.21.15.176:6812/3003473398 conn(0x7f16e403cea0 msgr2=0x7f16e403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 --2- 172.21.15.153:0/4098022291 >> v2:172.21.15.176:6812/3003473398 conn(0x7f16e403cea0 0x7f16e403f360 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7f16f80a12f0 tx=0x7f16e80a7000 comp rx=0 tx=0).stop 2024-09-10T20:28:48.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 10 monclient: shutdown 2024-09-10T20:28:48.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f80a1990 msgr2=0x7f16f8073cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 --2- 172.21.15.153:0/4098022291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f80a1990 0x7f16f8073cb0 secure :-1 s=READY pgs=8110 cs=0 l=1 rev1=1 crypto rx=0x7f16ec00c450 tx=0x7f16ec00c910 comp rx=0 tx=0).stop 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 shutdown_connections 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 --2- 172.21.15.153:0/4098022291 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f16f813cbf0 0x7f16f81ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 --2- 172.21.15.153:0/4098022291 >> v2:172.21.15.176:6812/3003473398 conn(0x7f16e403cea0 0x7f16e403f360 unknown :-1 s=CLOSED pgs=3254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 --2- 172.21.15.153:0/4098022291 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f16f81364f0 0x7f16f813c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 --2- 172.21.15.153:0/4098022291 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f16f80a1990 0x7f16f8073cb0 unknown :-1 s=CLOSED pgs=8110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 >> 172.21.15.153:0/4098022291 conn(0x7f16f80a71b0 msgr2=0x7f16f80a5090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 shutdown_connections 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 -- 172.21.15.153:0/4098022291 wait complete. 2024-09-10T20:28:48.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.054+0000 7f16fd02c640 1 librados: shutdown 2024-09-10T20:28:48.076 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T20:28:48.076 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T20:28:48.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T20:28:48.076 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:28:48.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.182+0000 7f391fab1640 10 monclient: get_monmap_and_config 2024-09-10T20:28:48.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.182+0000 7f391fab1640 10 monclient: build_initial_monmap 2024-09-10T20:28:48.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.182+0000 7f391fab1640 10 monclient: monmap: 2024-09-10T20:28:48.191 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:48.191 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.191 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:48.189379+0000 2024-09-10T20:28:48.191 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:48.189379+0000 2024-09-10T20:28:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:48.192 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:48.192 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 1 Processor -- start 2024-09-10T20:28:48.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 1 -- start start 2024-09-10T20:28:48.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 10 monclient: init 2024-09-10T20:28:48.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:48.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 10 monclient: _add_conns ranks=[1,4,0,3,2] 2024-09-10T20:28:48.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f39180a5ab0 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:48.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f39180a65a0 0x7f391813dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 10 monclient(hunting): picked mon.noname-c con 0x7f39180a65a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f391813e1b0 addr v1:172.21.15.153:6789/0 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3918070780 con 0x7f391813e1b0 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f39180709c0 con 0x7f39180a5ab0 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f39180a3010 con 0x7f39180a65a0 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391fab1640 10 monclient(hunting): authenticate will time out at 1088.329038s 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391e04e640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f391813e1b0 0x7f39181405a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:47054/0 (socket says 172.21.15.153:47054) 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391e04e640 1 -- 172.21.15.153:0/2418481978 learned_addr learned my addr 172.21.15.153:0/2418481978 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 1 --2- 172.21.15.153:0/2418481978 >> v2:172.21.15.176:3300/0 conn(0x7f39180a65a0 0x7f391813dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient(hunting): get_auth_request con 0x7f39180a65a0 auth_method 0 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient(hunting): handle_auth_done global_id 19678 payload 274 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 1 -- 172.21.15.153:0/2418481978 >> v1:172.21.15.176:6789/0 conn(0x7f39180a5ab0 legacy=0x7f39180a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 1 -- 172.21.15.153:0/2418481978 >> v1:172.21.15.153:6789/0 conn(0x7f391813e1b0 legacy=0x7f39181405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient: _finish_hunting 0 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 1 monclient: found mon.noname-c 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:28:48.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.186+0000 7f391d04c640 1 -- 172.21.15.153:0/2418481978 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39180a26e0 con 0x7f39180a65a0 2024-09-10T20:28:48.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391d04c640 1 -- 172.21.15.153:0/2418481978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39180a65a0 msgr2=0x7f391813dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391d04c640 1 --2- 172.21.15.153:0/2418481978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39180a65a0 0x7f391813dc70 secure :-1 s=READY pgs=5480 cs=0 l=1 rev1=1 crypto rx=0x7f39080097c0 tx=0x7f390801c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b6fa2e1169e70e70 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 1 -- 172.21.15.153:0/2418481978 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39080b0070 con 0x7f39180a65a0 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 10 monclient: dump: 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.199 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 10 monclient: _finish_auth 0 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 10 monclient: _check_auth_tickets 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 1 -- 172.21.15.153:0/2418481978 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f39080ab600 con 0x7f39180a65a0 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391fab1640 5 monclient: authenticate success, global_id 19678 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391fab1640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391fab1640 4 set_mon_vals no callback set 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 1 -- 172.21.15.153:0/2418481978 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39080ab9d0 con 0x7f39180a65a0 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391c84b640 10 monclient: dump: 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.201 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391fab1640 10 monclient: shutdown 2024-09-10T20:28:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391fab1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391fab1640 1 -- 172.21.15.153:0/2418481978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39180a65a0 msgr2=0x7f391813dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.190+0000 7f391fab1640 1 --2- 172.21.15.153:0/2418481978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39180a65a0 0x7f391813dc70 secure :-1 s=READY pgs=5480 cs=0 l=1 rev1=1 crypto rx=0x7f39080097c0 tx=0x7f390801c360 comp rx=0 tx=0).stop 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 -- 172.21.15.153:0/2418481978 shutdown_connections 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 --2- 172.21.15.153:0/2418481978 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39180a65a0 0x7f391813dc70 unknown :-1 s=CLOSED pgs=5480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 -- 172.21.15.153:0/2418481978 >> 172.21.15.153:0/2418481978 conn(0x7f39180aba00 msgr2=0x7f39180a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 -- 172.21.15.153:0/2418481978 shutdown_connections 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 -- 172.21.15.153:0/2418481978 wait complete. 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient: build_initial_monmap 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient: monmap: 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:48.207 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 librados: starting msgr at 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 librados: starting objecter 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 Processor -- start 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 -- start start 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 librados: setting wanted keys 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 librados: calling monclient init 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient: init 2024-09-10T20:28:48.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180a5ab0 0x7f39180a0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient(hunting): picked mon.noname-a con 0x7f39180a5ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39180a65a0 0x7f39181385b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient(hunting): picked mon.noname-b con 0x7f39180a65a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391813e1b0 0x7f39181d0730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient(hunting): picked mon.noname-c con 0x7f391813e1b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39180a3b70 con 0x7f39180a5ab0 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f39181444e0 con 0x7f39180a65a0 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f39180a90b0 con 0x7f391813e1b0 2024-09-10T20:28:48.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391fab1640 10 monclient(hunting): authenticate will time out at 1088.337951s 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180a5ab0 0x7f39180a0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391e04e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391813e1b0 0x7f39181d0730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180a5ab0 0x7f39180a0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54760/0 (socket says 172.21.15.153:54760) 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 1 -- 172.21.15.153:0/4172035229 learned_addr learned my addr 172.21.15.153:0/4172035229 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient(hunting): get_auth_request con 0x7f39180a5ab0 auth_method 0 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391e04e640 10 monclient(hunting): get_auth_request con 0x7f391813e1b0 auth_method 0 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391e04e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391e04e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391e04e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391e04e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391e04e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391e04e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d04c640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39180a65a0 0x7f39181385b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d04c640 10 monclient(hunting): get_auth_request con 0x7f39180a65a0 auth_method 0 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d04c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d04c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d04c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient(hunting): handle_auth_done global_id 26280 payload 824 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391813e1b0 msgr2=0x7f39181d0730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391813e1b0 0x7f39181d0730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39180a65a0 msgr2=0x7f39181385b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f39180a65a0 0x7f39181385b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient: _finish_hunting 0 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 1 monclient: found mon.noname-a 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391d84d640 1 -- 172.21.15.153:0/4172035229 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f391809f9b0 con 0x7f39180a5ab0 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.194+0000 7f391e04e640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f391813e1b0 0x7f39181d0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d84d640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180a5ab0 msgr2=0x7f39180a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d84d640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180a5ab0 0x7f39180a0b00 secure :-1 s=READY pgs=8112 cs=0 l=1 rev1=1 crypto rx=0x7f391400c450 tx=0x7f391400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 client.?.objecter ms_handle_connect 0x7f39180a5ab0 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914012a40 con 0x7f39180a5ab0 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 monclient: dump: 2024-09-10T20:28:48.212 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 monclient: _finish_auth 0 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3914017070 con 0x7f39180a5ab0 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 5 monclient: authenticate success, global_id 26280 2024-09-10T20:28:48.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 10 monclient: _renew_subs 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3918121c50 con 0x7f39180a5ab0 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3914012e40 con 0x7f39180a5ab0 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 10 monclient: dump: 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.214 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390f7fe640 4 set_mon_vals no callback set 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 10 client.26280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 10 monclient: _renew_subs 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39180a1490 con 0x7f39180a5ab0 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 1 librados: init done 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391809e9f0 0x7f39181d0c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39180a20e0 con 0x7f391809e9f0 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391809e9f0 0x7f39181d0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 10 monclient: get_auth_request con 0x7f391809e9f0 auth_method 0 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d84d640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180a5ab0 0x7f39180a0b00 secure :-1 s=READY pgs=8112 cs=0 l=1 rev1=1 crypto rx=0x7f391400c450 tx=0x7f391400c910 comp rx=0 tx=0).queue_received will delay after 1.726e+09 on 0x7f391401c460 mgrmap(e 5) 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f391401c460 con 0x7f39180a5ab0 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 10 monclient(hunting): handle_auth_done global_id 26283 payload 824 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391d04c640 1 -- 172.21.15.153:0/4172035229 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f39180a5e90 con 0x7f391809e9f0 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f390e7fc640 1 --2- 172.21.15.153:0/4172035229 >> v2:172.21.15.176:6812/3003473398 conn(0x7f38fc03cea0 0x7f38fc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391e04e640 1 --2- 172.21.15.153:0/4172035229 >> v2:172.21.15.176:6812/3003473398 conn(0x7f38fc03cea0 0x7f38fc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.198+0000 7f391e04e640 10 monclient: get_auth_request con 0x7f38fc03cea0 auth_method 0 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 10 client.26280.objecter ms_handle_connect 0x7f391809e9f0 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f391d04c640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391809e9f0 msgr2=0x7f39181d0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 10 client.26280.objecter resend_mon_ops 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f391d04c640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391809e9f0 0x7f39181d0c70 secure :-1 s=READY pgs=8113 cs=0 l=1 rev1=1 crypto rx=0x7f39180a3430 tx=0x7f3908019100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39080bb070 con 0x7f391809e9f0 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 10 monclient: dump: 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.219 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f39080b6440 con 0x7f391809e9f0 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391809e9f0 msgr2=0x7f39181d0c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391809e9f0 0x7f39181d0c70 secure :-1 s=READY pgs=8113 cs=0 l=1 rev1=1 crypto rx=0x7f39180a3430 tx=0x7f3908019100 comp rx=0 tx=0).stop 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391809e9f0 msgr2=0x7f39181d0c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:48.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f391809e9f0 0x7f39181d0c70 secure :-1 s=CLOSED pgs=8113 cs=0 l=1 rev1=1 crypto rx=0x7f39180a3430 tx=0x7f3908019100 comp rx=0 tx=0).stop 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f390e7fc640 10 client.26280.objecter ms_handle_connect 0x7f38fc03cea0 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.202+0000 7f391e04e640 1 --2- 172.21.15.153:0/4172035229 >> v2:172.21.15.176:6812/3003473398 conn(0x7f38fc03cea0 0x7f38fc03f360 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f3910005c10 tx=0x7f3910009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391fab1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391fab1640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39181d11b0 0x7f391813bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391fab1640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f39181d1590 con 0x7f39181d11b0 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39181d11b0 0x7f391813bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 10 monclient: get_auth_request con 0x7f39181d11b0 auth_method 0 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 10 monclient(hunting): handle_auth_done global_id 26286 payload 824 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.206+0000 7f391d04c640 1 -- 172.21.15.153:0/4172035229 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f39180a6be0 con 0x7f39181d11b0 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 10 client.26280.objecter ms_handle_connect 0x7f39181d11b0 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 10 client.26280.objecter resend_mon_ops 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391d04c640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39181d11b0 msgr2=0x7f391813bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391d04c640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39181d11b0 0x7f391813bc60 secure :-1 s=READY pgs=8114 cs=0 l=1 rev1=1 crypto rx=0x7f39080ae040 tx=0x7f3908008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39080b0040 con 0x7f39181d11b0 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 10 monclient: dump: 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.222 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4555 (secure 0 0 0) 0x7f39080ca430 con 0x7f39181d11b0 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39181d11b0 msgr2=0x7f391813bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39181d11b0 0x7f391813bc60 secure :-1 s=READY pgs=8114 cs=0 l=1 rev1=1 crypto rx=0x7f39080ae040 tx=0x7f3908008040 comp rx=0 tx=0).stop 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39181d11b0 msgr2=0x7f391813bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:48.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39181d11b0 0x7f391813bc60 secure :-1 s=CLOSED pgs=8114 cs=0 l=1 rev1=1 crypto rx=0x7f39080ae040 tx=0x7f3908008040 comp rx=0 tx=0).stop 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 174899, 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:28:48.224 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:48.225 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:48.226 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:28:48.227 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:48.228 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.229 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.230 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:48.231 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:28:48.232 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:48.234 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:48.235 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:28:48.236 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.236 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.236 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:28:48.236 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 10 librados: watch_flush enter 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 10 librados: watch_flush exit 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 20 client.26280.objecter shutdown clearing up homeless session... 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 10 client.26280.objecter successfully canceled tick 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 >> v2:172.21.15.176:6812/3003473398 conn(0x7f38fc03cea0 msgr2=0x7f38fc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 1 --2- 172.21.15.153:0/4172035229 >> v2:172.21.15.176:6812/3003473398 conn(0x7f38fc03cea0 0x7f38fc03f360 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f3910005c10 tx=0x7f3910009040 comp rx=0 tx=0).stop 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 10 monclient: shutdown 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180a5ab0 msgr2=0x7f39180a0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 1 --2- 172.21.15.153:0/4172035229 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f39180a5ab0 0x7f39180a0b00 secure :-1 s=READY pgs=8112 cs=0 l=1 rev1=1 crypto rx=0x7f391400c450 tx=0x7f391400c910 comp rx=0 tx=0).stop 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391e04e640 1 -- 172.21.15.153:0/4172035229 reap_dead start 2024-09-10T20:28:48.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 1 -- 172.21.15.153:0/4172035229 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6016+0+0 (unknown 0 0 0) 0x7f391404f910 con 0x7f39180a5ab0 2024-09-10T20:28:48.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 10 client.26280.objecter ms_dispatch 0x7f3918000ba0 osd_map(103..103 src has 1..103) 2024-09-10T20:28:48.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391e04e640 1 -- 172.21.15.153:0/4172035229 reap_dead start 2024-09-10T20:28:48.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f390e7fc640 10 librados: disconnected, discarding osd_map(103..103 src has 1..103) 2024-09-10T20:28:48.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 shutdown_connections 2024-09-10T20:28:48.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 >> 172.21.15.153:0/4172035229 conn(0x7f39180aba00 msgr2=0x7f39180aa7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:48.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 shutdown_connections 2024-09-10T20:28:48.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 1 -- 172.21.15.153:0/4172035229 wait complete. 2024-09-10T20:28:48.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.210+0000 7f391fab1640 1 librados: shutdown 2024-09-10T20:28:48.239 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:28:48.239 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T20:28:48.240 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T20:28:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.346+0000 7fb6dfd56640 10 monclient: get_monmap_and_config 2024-09-10T20:28:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.346+0000 7fb6dfd56640 10 monclient: build_initial_monmap 2024-09-10T20:28:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.346+0000 7fb6dfd56640 10 monclient: monmap: 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:48.353599+0000 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:48.353599+0000 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:48.356 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 1 Processor -- start 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 1 -- start start 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient: init 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient: _add_conns ranks=[4,0,2,1,3] 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fb6d81364f0 0x7fb6d81368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d81364f0 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb6d81360a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d80a1990 0x7fb6d809d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient(hunting): picked mon.noname-b con 0x7fb6d80a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6d80a27a0 con 0x7fb6d81360a0 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fb6d80a2a80 con 0x7fb6d81364f0 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d80a2d60 con 0x7fb6d80a1990 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dfd56640 10 monclient(hunting): authenticate will time out at 1088.493613s 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dd2f1640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb6d81360a0 0x7fb6d809ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:47058/0 (socket says 172.21.15.153:47058) 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6dd2f1640 1 -- 172.21.15.153:0/1378148654 learned_addr learned my addr 172.21.15.153:0/1378148654 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 1 --2- 172.21.15.153:0/1378148654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d80a1990 0x7fb6d809d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient(hunting): get_auth_request con 0x7fb6d80a1990 auth_method 0 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6ddaf2640 1 --2- 172.21.15.153:0/1378148654 >> v2:172.21.15.176:3300/0 conn(0x7fb6d81364f0 0x7fb6d81368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6ddaf2640 10 monclient(hunting): get_auth_request con 0x7fb6d81364f0 auth_method 0 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6ddaf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6ddaf2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6ddaf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient(hunting): handle_auth_done global_id 24203 payload 274 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 1 -- 172.21.15.153:0/1378148654 >> v2:172.21.15.176:3300/0 conn(0x7fb6d81364f0 msgr2=0x7fb6d81368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 1 --2- 172.21.15.153:0/1378148654 >> v2:172.21.15.176:3300/0 conn(0x7fb6d81364f0 0x7fb6d81368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 1 -- 172.21.15.153:0/1378148654 >> v1:172.21.15.153:6789/0 conn(0x7fb6d81360a0 legacy=0x7fb6d809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient: _finish_hunting 0 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 1 monclient: found mon.noname-b 2024-09-10T20:28:48.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:48.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:28:48.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.350+0000 7fb6de2f3640 1 -- 172.21.15.153:0/1378148654 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d8136e10 con 0x7fb6d80a1990 2024-09-10T20:28:48.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6de2f3640 1 -- 172.21.15.153:0/1378148654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d80a1990 msgr2=0x7fb6d809d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6de2f3640 1 --2- 172.21.15.153:0/1378148654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d80a1990 0x7fb6d809d520 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0009d90 tx=0x7fb6d001c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=17291680b868333 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 1 -- 172.21.15.153:0/1378148654 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d00b2070 con 0x7fb6d80a1990 2024-09-10T20:28:48.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 10 monclient: dump: 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 10 monclient: _finish_auth 0 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 10 monclient: _check_auth_tickets 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:48.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 1 -- 172.21.15.153:0/1378148654 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d809dc70 con 0x7fb6d80a1990 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 1 -- 172.21.15.153:0/1378148654 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d00a5ab0 con 0x7fb6d80a1990 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dcaf0640 10 monclient: dump: 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.367 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dfd56640 5 monclient: authenticate success, global_id 24203 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dfd56640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dfd56640 4 set_mon_vals no callback set 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dfd56640 10 monclient: shutdown 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dfd56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dfd56640 1 -- 172.21.15.153:0/1378148654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d80a1990 msgr2=0x7fb6d809d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.354+0000 7fb6dfd56640 1 --2- 172.21.15.153:0/1378148654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d80a1990 0x7fb6d809d520 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb6d0009d90 tx=0x7fb6d001c830 comp rx=0 tx=0).stop 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 -- 172.21.15.153:0/1378148654 shutdown_connections 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 --2- 172.21.15.153:0/1378148654 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d80a1990 0x7fb6d809d520 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 --2- 172.21.15.153:0/1378148654 >> v2:172.21.15.176:3300/0 conn(0x7fb6d81364f0 0x7fb6d81368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 -- 172.21.15.153:0/1378148654 >> 172.21.15.153:0/1378148654 conn(0x7fb6d80a71b0 msgr2=0x7fb6d80a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 -- 172.21.15.153:0/1378148654 shutdown_connections 2024-09-10T20:28:48.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 -- 172.21.15.153:0/1378148654 wait complete. 2024-09-10T20:28:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient: build_initial_monmap 2024-09-10T20:28:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient: monmap: 2024-09-10T20:28:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 librados: starting msgr at 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 librados: starting objecter 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 Processor -- start 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 -- start start 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 librados: setting wanted keys 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 librados: calling monclient init 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient: init 2024-09-10T20:28:48.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d80a1990 0x7fb6d81d0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient(hunting): picked mon.noname-a con 0x7fb6d80a1990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d81360a0 0x7fb6d81d0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d81360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6d81364f0 0x7fb6d81d11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient(hunting): picked mon.noname-b con 0x7fb6d81364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d8140ff0 con 0x7fb6d80a1990 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb6d8140e70 con 0x7fb6d81364f0 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d8141170 con 0x7fb6d81360a0 2024-09-10T20:28:48.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dfd56640 10 monclient(hunting): authenticate will time out at 1088.503571s 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dd2f1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d81360a0 0x7fb6d81d0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d80a1990 0x7fb6d81d0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d80a1990 0x7fb6d81d0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54796/0 (socket says 172.21.15.153:54796) 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dd2f1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d81360a0 0x7fb6d81d0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37106/0 (socket says 172.21.15.153:37106) 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 1 -- 172.21.15.153:0/4209700944 learned_addr learned my addr 172.21.15.153:0/4209700944 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 10 monclient(hunting): get_auth_request con 0x7fb6d80a1990 auth_method 0 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dd2f1640 10 monclient(hunting): get_auth_request con 0x7fb6d81360a0 auth_method 0 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dd2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dd2f1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dd2f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6de2f3640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6d81364f0 0x7fb6d81d11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6ddaf2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.358+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6de2f3640 10 monclient(hunting): get_auth_request con 0x7fb6d81364f0 auth_method 0 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6de2f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6de2f3640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6de2f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 10 monclient(hunting): handle_auth_done global_id 26289 payload 824 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d81360a0 msgr2=0x7fb6d81d0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d81360a0 0x7fb6d81d0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6d81364f0 msgr2=0x7fb6d81d11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6d81364f0 0x7fb6d81d11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 10 monclient: _finish_hunting 0 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 1 monclient: found mon.noname-a 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 1 -- 172.21.15.153:0/4209700944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6d813d380 con 0x7fb6d80a1990 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6de2f3640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb6d81364f0 0x7fb6d81d11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d80a1990 msgr2=0x7fb6d81d0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ddaf2640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d80a1990 0x7fb6d81d0770 secure :-1 s=READY pgs=8116 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400d800 tx=0x7fb6d400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 client.?.objecter ms_handle_connect 0x7fb6d80a1990 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4018020 con 0x7fb6d80a1990 2024-09-10T20:28:48.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 monclient: dump: 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 monclient: _finish_auth 0 2024-09-10T20:28:48.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6d40098d0 con 0x7fb6d80a1990 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6d4009cc0 con 0x7fb6d80a1990 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 5 monclient: authenticate success, global_id 26289 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 10 monclient: _renew_subs 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6d813c180 con 0x7fb6d80a1990 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:48.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6cffff640 4 set_mon_vals no callback set 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6ce7fc640 10 monclient: dump: 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 10 client.26289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 10 monclient: _renew_subs 2024-09-10T20:28:48.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6d813d0a0 con 0x7fb6d80a1990 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 1 librados: init done 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.362+0000 7fb6dfd56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dfd56640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d813db00 0x7fb6d812f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dfd56640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d809dc70 con 0x7fb6d813db00 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d813db00 0x7fb6d812f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 10 monclient: get_auth_request con 0x7fb6d813db00 auth_method 0 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_done global_id 24209 payload 824 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 1 -- 172.21.15.153:0/4209700944 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb6d813cc70 con 0x7fb6d813db00 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fb6d401b020 con 0x7fb6d80a1990 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 1 --2- 172.21.15.153:0/4209700944 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb6bc03cea0 0x7fb6bc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 6016+0+0 (secure 0 0 0) 0x7fb6d4050180 con 0x7fb6d80a1990 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 10 client.26289.objecter ms_dispatch 0x7fb6d8000ba0 osd_map(103..103 src has 1..103) 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 3 client.26289.objecter handle_osd_map got epochs [103,103] > 0 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 3 client.26289.objecter handle_osd_map decoding full epoch 103 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 20 client.26289.objecter dump_active .. 0 homeless 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6de2f3640 1 --2- 172.21.15.153:0/4209700944 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb6bc03cea0 0x7fb6bc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6de2f3640 10 monclient: get_auth_request con 0x7fb6bc03cea0 auth_method 0 2024-09-10T20:28:48.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 10 client.26289.objecter ms_handle_connect 0x7fb6d813db00 2024-09-10T20:28:48.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d813db00 msgr2=0x7fb6d812f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 10 client.26289.objecter resend_mon_ops 2024-09-10T20:28:48.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6dd2f1640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d813db00 0x7fb6d812f520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800ea70 tx=0x7fb6c800eb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c800c840 con 0x7fb6d813db00 2024-09-10T20:28:48.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:48.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 10 monclient: dump: 2024-09-10T20:28:48.386 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.386 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.386 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb6c8019070 con 0x7fb6d813db00 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d813db00 msgr2=0x7fb6d812f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d813db00 0x7fb6d812f520 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800ea70 tx=0x7fb6c800eb60 comp rx=0 tx=0).stop 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d813db00 msgr2=0x7fb6d812f520 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:48.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.366+0000 7fb6ce7fc640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d813db00 0x7fb6d812f520 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb6c800ea70 tx=0x7fb6c800eb60 comp rx=0 tx=0).stop 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.370+0000 7fb6de2f3640 1 --2- 172.21.15.153:0/4209700944 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb6bc03cea0 0x7fb6bc03f360 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00ac000 tx=0x7fb6d00a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.370+0000 7fb6ce7fc640 10 client.26289.objecter ms_handle_connect 0x7fb6bc03cea0 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dfd56640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dfd56640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d812fa60 0x7fb6d812fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dfd56640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb6d8130380 con 0x7fb6d812fa60 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d812fa60 0x7fb6d812fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 10 monclient: get_auth_request con 0x7fb6d812fa60 auth_method 0 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 10 monclient(hunting): handle_auth_done global_id 24212 payload 824 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.374+0000 7fb6dd2f1640 1 -- 172.21.15.153:0/4209700944 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6d81368d0 con 0x7fb6d812fa60 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dd2f1640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d812fa60 msgr2=0x7fb6d812fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 10 client.26289.objecter ms_handle_connect 0x7fb6d812fa60 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 10 client.26289.objecter resend_mon_ops 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dd2f1640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d812fa60 0x7fb6d812fe40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8007930 tx=0x7fb6c8004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6c801ec50 con 0x7fb6d812fa60 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 10 monclient: dump: 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.389 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fb6c801ec50 con 0x7fb6d812fa60 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d812fa60 msgr2=0x7fb6d812fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d812fa60 0x7fb6d812fe40 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8007930 tx=0x7fb6c8004950 comp rx=0 tx=0).stop 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d812fa60 msgr2=0x7fb6d812fe40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:48.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6ce7fc640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb6d812fa60 0x7fb6d812fe40 secure :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb6c8007930 tx=0x7fb6c8004950 comp rx=0 tx=0).stop 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 29947, 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.391 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:48.392 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:48.393 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:28:48.394 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:48.395 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:48.396 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:48.397 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:48.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.398 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:28:48.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:28:48.398 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:48.398 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:48.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.398 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:48.398 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:48.399 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:28:48.400 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:28:48.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:28:48.402 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:28:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 10 librados: watch_flush enter 2024-09-10T20:28:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 10 librados: watch_flush exit 2024-09-10T20:28:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 20 client.26289.objecter shutdown clearing up homeless session... 2024-09-10T20:28:48.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 10 client.26289.objecter successfully canceled tick 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb6bc03cea0 msgr2=0x7fb6bc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 --2- 172.21.15.153:0/4209700944 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb6bc03cea0 0x7fb6bc03f360 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7fb6d00ac000 tx=0x7fb6d00a7070 comp rx=0 tx=0).stop 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 10 monclient: shutdown 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d80a1990 msgr2=0x7fb6d81d0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d80a1990 0x7fb6d81d0770 secure :-1 s=READY pgs=8116 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400d800 tx=0x7fb6d400dcc0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6de2f3640 1 -- 172.21.15.153:0/4209700944 reap_dead start 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 shutdown_connections 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 --2- 172.21.15.153:0/4209700944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb6d80a1990 0x7fb6d81d0770 unknown :-1 s=CLOSED pgs=8116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 >> 172.21.15.153:0/4209700944 conn(0x7fb6d80a71b0 msgr2=0x7fb6d80a5800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 shutdown_connections 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 -- 172.21.15.153:0/4209700944 wait complete. 2024-09-10T20:28:48.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.378+0000 7fb6dfd56640 1 librados: shutdown 2024-09-10T20:28:48.406 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:28:48.406 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T20:28:48.406 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:28:48.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient: get_monmap_and_config 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient: build_initial_monmap 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient: monmap: 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:48.526936+0000 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:48.526936+0000 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:48.529 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:48.530 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:48.530 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 1 Processor -- start 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 1 -- start start 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient: init 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0c201364f0 0x7f0c201368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient(hunting): picked mon.noname-c con 0x7f0c201364f0 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2009ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient(hunting): picked mon.noname-b con 0x7f0c201360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0c200a1990 0x7f0c2009d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient(hunting): picked mon.noname-a con 0x7f0c200a1990 addr v2:172.21.15.153:3300/0 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f0c200a27a0 con 0x7f0c200a1990 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0c200a2a80 con 0x7f0c201364f0 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0c200a2d60 con 0x7f0c201360a0 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c25138640 10 monclient(hunting): authenticate will time out at 1088.666464s 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2009ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0c200a1990 0x7f0c2009d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2009ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37134/0 (socket says 172.21.15.153:37134) 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 1 -- 172.21.15.153:0/4122458030 learned_addr learned my addr 172.21.15.153:0/4122458030 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 10 monclient(hunting): get_auth_request con 0x7f0c201360a0 auth_method 0 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient(hunting): get_auth_request con 0x7f0c200a1990 auth_method 0 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1ed9d640 1 --2- 172.21.15.153:0/4122458030 >> v2:172.21.15.176:3300/0 conn(0x7f0c201364f0 0x7f0c201368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1ed9d640 10 monclient(hunting): get_auth_request con 0x7f0c201364f0 auth_method 0 2024-09-10T20:28:48.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient(hunting): handle_auth_done global_id 26292 payload 274 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 1 -- 172.21.15.153:0/4122458030 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 msgr2=0x7f0c2009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 1 --2- 172.21.15.153:0/4122458030 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2009ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 1 -- 172.21.15.153:0/4122458030 >> v2:172.21.15.176:3300/0 conn(0x7f0c201364f0 msgr2=0x7f0c201368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 1 --2- 172.21.15.153:0/4122458030 >> v2:172.21.15.176:3300/0 conn(0x7f0c201364f0 0x7f0c201368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient: _finish_hunting 0 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 1 monclient: found mon.noname-a 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1f59e640 1 -- 172.21.15.153:0/4122458030 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c20136e10 con 0x7f0c200a1990 2024-09-10T20:28:48.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.522+0000 7f0c1e59c640 1 --2- 172.21.15.153:0/4122458030 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2009ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1f59e640 1 -- 172.21.15.153:0/4122458030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 msgr2=0x7f0c2009d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1f59e640 1 --2- 172.21.15.153:0/4122458030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 0x7f0c2009d350 secure :-1 s=READY pgs=8119 cs=0 l=1 rev1=1 crypto rx=0x7f0c200a12f0 tx=0x7f0c1801c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9353be1a707ee5d3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 1 -- 172.21.15.153:0/4122458030 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c180b2070 con 0x7f0c200a1990 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 10 monclient: dump: 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.539 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 10 monclient: _finish_auth 0 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 10 monclient: _check_auth_tickets 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 1 -- 172.21.15.153:0/4122458030 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c180a5810 con 0x7f0c200a1990 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c25138640 5 monclient: authenticate success, global_id 26292 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c25138640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c25138640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 1 -- 172.21.15.153:0/4122458030 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c180a5be0 con 0x7f0c200a1990 2024-09-10T20:28:48.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c25138640 4 set_mon_vals no callback set 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c1dd9b640 10 monclient: dump: 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c25138640 10 monclient: shutdown 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c25138640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c25138640 1 -- 172.21.15.153:0/4122458030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 msgr2=0x7f0c2009d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.526+0000 7f0c25138640 1 --2- 172.21.15.153:0/4122458030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 0x7f0c2009d350 secure :-1 s=READY pgs=8119 cs=0 l=1 rev1=1 crypto rx=0x7f0c200a12f0 tx=0x7f0c1801c880 comp rx=0 tx=0).stop 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 -- 172.21.15.153:0/4122458030 shutdown_connections 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 --2- 172.21.15.153:0/4122458030 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 0x7f0c2009d350 unknown :-1 s=CLOSED pgs=8119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 --2- 172.21.15.153:0/4122458030 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2009ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 --2- 172.21.15.153:0/4122458030 >> v2:172.21.15.176:3300/0 conn(0x7f0c201364f0 0x7f0c201368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 -- 172.21.15.153:0/4122458030 >> 172.21.15.153:0/4122458030 conn(0x7f0c200a71b0 msgr2=0x7f0c200a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 -- 172.21.15.153:0/4122458030 shutdown_connections 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 -- 172.21.15.153:0/4122458030 wait complete. 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient: build_initial_monmap 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient: monmap: 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:48.542 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 librados: starting msgr at 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 librados: starting objecter 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 Processor -- start 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 -- start start 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 librados: setting wanted keys 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 librados: calling monclient init 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient: init 2024-09-10T20:28:48.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 0x7f0c201ce2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient(hunting): picked mon.noname-a con 0x7f0c200a1990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2013c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient(hunting): picked mon.noname-c con 0x7f0c201360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c201364f0 0x7f0c2013c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient(hunting): picked mon.noname-b con 0x7f0c201364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2013ffe0 con 0x7f0c200a1990 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2013fe60 con 0x7f0c201364f0 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0c20140160 con 0x7f0c201360a0 2024-09-10T20:28:48.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c25138640 10 monclient(hunting): authenticate will time out at 1088.673563s 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 0x7f0c201ce2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2013c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 0x7f0c201ce2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:54816/0 (socket says 172.21.15.153:54816) 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 1 -- 172.21.15.153:0/3612053355 learned_addr learned my addr 172.21.15.153:0/3612053355 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient(hunting): get_auth_request con 0x7f0c200a1990 auth_method 0 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2013c340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37142/0 (socket says 172.21.15.153:37142) 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 10 monclient(hunting): get_auth_request con 0x7f0c201360a0 auth_method 0 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient(hunting): handle_auth_done global_id 26295 payload 824 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 msgr2=0x7f0c2013c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2013c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c201364f0 msgr2=0x7f0c2013c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1f59e640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c201364f0 0x7f0c2013c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c201364f0 0x7f0c2013c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient: _finish_hunting 0 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 1 monclient: found mon.noname-a 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:48.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1ed9d640 1 -- 172.21.15.153:0/3612053355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c201ce810 con 0x7f0c200a1990 2024-09-10T20:28:48.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.530+0000 7f0c1e59c640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0c201360a0 0x7f0c2013c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:28:48.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1ed9d640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 msgr2=0x7f0c201ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1ed9d640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 0x7f0c201ce2d0 secure :-1 s=READY pgs=8120 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000d830 tx=0x7f0c1000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 client.?.objecter ms_handle_connect 0x7f0c200a1990 2024-09-10T20:28:48.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:48.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1000b8d0 con 0x7f0c200a1990 2024-09-10T20:28:48.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:28:48.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 monclient: dump: 2024-09-10T20:28:48.550 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 monclient: _finish_auth 0 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 monclient: _check_auth_tickets 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:48.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c10019070 con 0x7f0c200a1990 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 5 monclient: authenticate success, global_id 26295 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 10 monclient: _renew_subs 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c20139610 con 0x7f0c200a1990 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c1000bc70 con 0x7f0c200a1990 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 10 client.26295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 monclient: dump: 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.552 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1d59a640 4 set_mon_vals no callback set 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 10 monclient: _renew_subs 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c2013a570 con 0x7f0c200a1990 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 1 librados: init done 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137370 0x7f0c20137750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0c2009e0a0 con 0x7f0c20137370 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f0c1001e460 con 0x7f0c200a1990 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137370 0x7f0c20137750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 10 monclient: get_auth_request con 0x7f0c20137370 auth_method 0 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 1 --2- 172.21.15.153:0/3612053355 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0c0c03cea0 0x7f0c0c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 6775+0+0 (secure 0 0 0) 0x7f0c10050330 con 0x7f0c200a1990 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 10 client.26295.objecter ms_dispatch 0x7f0c20000ba0 osd_map(104..104 src has 1..104) 2024-09-10T20:28:48.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 3 client.26295.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 3 client.26295.objecter handle_osd_map decoding full epoch 104 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c077fe640 20 client.26295.objecter dump_active .. 0 homeless 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_done global_id 19693 payload 824 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1e59c640 1 -- 172.21.15.153:0/3612053355 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0c2013acc0 con 0x7f0c20137370 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1f59e640 1 --2- 172.21.15.153:0/3612053355 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0c0c03cea0 0x7f0c0c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.534+0000 7f0c1f59e640 10 monclient: get_auth_request con 0x7f0c0c03cea0 auth_method 0 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c1e59c640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137370 msgr2=0x7f0c20137750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 10 client.26295.objecter ms_handle_connect 0x7f0c20137370 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 10 client.26295.objecter resend_mon_ops 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c1e59c640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137370 0x7f0c20137750 secure :-1 s=READY pgs=5483 cs=0 l=1 rev1=1 crypto rx=0x7f0c2013d1e0 tx=0x7f0c1400f610 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14019070 con 0x7f0c20137370 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:28:48.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 10 monclient: dump: 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 10 client.26295.objecter ms_handle_connect 0x7f0c0c03cea0 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c1f59e640 1 --2- 172.21.15.153:0/3612053355 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0c0c03cea0 0x7f0c0c03f360 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7f0c180ac000 tx=0x7f0c180a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0c14004240 con 0x7f0c20137370 2024-09-10T20:28:48.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:48.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:48.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137370 msgr2=0x7f0c20137750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137370 0x7f0c20137750 secure :-1 s=READY pgs=5483 cs=0 l=1 rev1=1 crypto rx=0x7f0c2013d1e0 tx=0x7f0c1400f610 comp rx=0 tx=0).stop 2024-09-10T20:28:48.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137370 msgr2=0x7f0c20137750 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:48.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.538+0000 7f0c077fe640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137370 0x7f0c20137750 unknown :-1 s=CLOSED pgs=5483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.542+0000 7f0c25138640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.542+0000 7f0c25138640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137c90 0x7f0c2006f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.542+0000 7f0c25138640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.542+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0c201368d0 con 0x7f0c20137c90 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137c90 0x7f0c2006f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 10 monclient: get_auth_request con 0x7f0c20137c90 auth_method 0 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 10 monclient(hunting): handle_auth_done global_id 19696 payload 824 2024-09-10T20:28:48.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 1 -- 172.21.15.153:0/3612053355 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0c2006a660 con 0x7f0c20137c90 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137c90 msgr2=0x7f0c2006f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c1e59c640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137c90 0x7f0c2006f830 secure :-1 s=READY pgs=5484 cs=0 l=1 rev1=1 crypto rx=0x7f0c1401b000 tx=0x7f0c140046d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c077fe640 10 client.26295.objecter ms_handle_connect 0x7f0c20137c90 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c077fe640 10 client.26295.objecter resend_mon_ops 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c14019040 con 0x7f0c20137c90 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c077fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.546+0000 7f0c077fe640 10 monclient: dump: 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.560 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f0c1401e950 con 0x7f0c20137c90 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c077fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c077fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137c90 msgr2=0x7f0c2006f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c077fe640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137c90 0x7f0c2006f830 secure :-1 s=READY pgs=5484 cs=0 l=1 rev1=1 crypto rx=0x7f0c1401b000 tx=0x7f0c140046d0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c077fe640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137c90 msgr2=0x7f0c2006f830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c077fe640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0c20137c90 0x7f0c2006f830 secure :-1 s=CLOSED pgs=5484 cs=0 l=1 rev1=1 crypto rx=0x7f0c1401b000 tx=0x7f0c140046d0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.561 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 18, 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 175196, 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 5, 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:28:48.562 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:28:48.563 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:28:48.564 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:28:48.565 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:28:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:28:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:28:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:28:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:28:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:28:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:28:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:28:48.566 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:28:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:28:48.567 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:48.568 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:28:48.569 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.570 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.571 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.572 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:48.573 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:48.574 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 8 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:28:48.575 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:28:48.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 10 librados: watch_flush enter 2024-09-10T20:28:48.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 10 librados: watch_flush exit 2024-09-10T20:28:48.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 20 client.26295.objecter shutdown clearing up homeless session... 2024-09-10T20:28:48.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 10 client.26295.objecter successfully canceled tick 2024-09-10T20:28:48.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0c0c03cea0 msgr2=0x7f0c0c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 1 --2- 172.21.15.153:0/3612053355 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0c0c03cea0 0x7f0c0c03f360 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7f0c180ac000 tx=0x7f0c180a7070 comp rx=0 tx=0).stop 2024-09-10T20:28:48.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 10 monclient: shutdown 2024-09-10T20:28:48.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:48.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 msgr2=0x7f0c201ce2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 1 --2- 172.21.15.153:0/3612053355 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0c200a1990 0x7f0c201ce2d0 secure :-1 s=READY pgs=8120 cs=0 l=1 rev1=1 crypto rx=0x7f0c1000d830 tx=0x7f0c1000dc30 comp rx=0 tx=0).stop 2024-09-10T20:28:48.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c1f59e640 1 -- 172.21.15.153:0/3612053355 reap_dead start 2024-09-10T20:28:48.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c1f59e640 1 -- 172.21.15.153:0/3612053355 reap_dead start 2024-09-10T20:28:48.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 shutdown_connections 2024-09-10T20:28:48.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 >> 172.21.15.153:0/3612053355 conn(0x7f0c200a71b0 msgr2=0x7f0c200a56f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:48.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 shutdown_connections 2024-09-10T20:28:48.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 1 -- 172.21.15.153:0/3612053355 wait complete. 2024-09-10T20:28:48.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.550+0000 7f0c25138640 1 librados: shutdown 2024-09-10T20:28:48.578 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:28:48.578 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-10T20:28:48.579 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T20:28:48.579 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-10T20:28:48.579 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-10T20:28:48.579 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T20:28:48.579 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:28:48.591 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T20:28:48.591 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T20:28:48.592 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:28:48.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient: get_monmap_and_config 2024-09-10T20:28:48.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient: build_initial_monmap 2024-09-10T20:28:48.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient: monmap: 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:48.747671+0000 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:48.747671+0000 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:48.750 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 1 Processor -- start 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 1 -- start start 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient: init 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient: _add_conns ranks=[3,2,1,0,4] 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7cf81364f0 0x7f7cf81368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient(hunting): picked mon.noname-a con 0x7f7cf81364f0 addr v2:172.21.15.153:3300/0 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient(hunting): picked mon.noname-b con 0x7f7cf81360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7cf80a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cf80a27a0 con 0x7f7cf80a1990 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f7cf80a2a80 con 0x7f7cf81364f0 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf80a2d60 con 0x7f7cf81360a0 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7d004b2640 10 monclient(hunting): authenticate will time out at 1088.887315s 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfe24e640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f7cf81364f0 msgr2=0x7f7cf81368d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfe24e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7cf81364f0 0x7f7cf81368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf809ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37152/0 (socket says 172.21.15.153:37152) 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 1 -- 172.21.15.153:0/783861416 learned_addr learned my addr 172.21.15.153:0/783861416 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient(hunting): get_auth_request con 0x7f7cf81360a0 auth_method 0 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient(hunting): handle_auth_done global_id 24221 payload 274 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 1 -- 172.21.15.153:0/783861416 >> v2:172.21.15.153:3300/0 conn(0x7f7cf81364f0 msgr2=0x7f7cf81368d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 1 --2- 172.21.15.153:0/783861416 >> v2:172.21.15.153:3300/0 conn(0x7f7cf81364f0 0x7f7cf81368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 1 -- 172.21.15.153:0/783861416 >> v1:172.21.15.176:6789/0 conn(0x7f7cf80a1990 legacy=0x7f7cf809d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient: _finish_hunting 0 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 1 monclient: found mon.noname-b 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:28:48.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.742+0000 7f7cfda4d640 1 -- 172.21.15.153:0/783861416 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf8136e10 con 0x7f7cf81360a0 2024-09-10T20:28:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.746+0000 7f7cfda4d640 1 -- 172.21.15.153:0/783861416 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 msgr2=0x7f7cf809ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.746+0000 7f7cfda4d640 1 --2- 172.21.15.153:0/783861416 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf809ce10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80a12f0 tx=0x7f7ce801c960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a2ad8fd7d6dd4dd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 1 -- 172.21.15.153:0/783861416 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80a56b0 con 0x7f7cf81360a0 2024-09-10T20:28:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 10 monclient: dump: 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 10 monclient: _finish_auth 0 2024-09-10T20:28:48.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 10 monclient: _check_auth_tickets 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 1 -- 172.21.15.153:0/783861416 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce80a5850 con 0x7f7cf81360a0 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 5 monclient: authenticate success, global_id 24221 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 4 set_mon_vals no callback set 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 1 -- 172.21.15.153:0/783861416 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80a5c20 con 0x7f7cf81360a0 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfd24c640 10 monclient: dump: 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.761 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient: shutdown 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 -- 172.21.15.153:0/783861416 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 msgr2=0x7f7cf809ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 --2- 172.21.15.153:0/783861416 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf809ce10 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f7cf80a12f0 tx=0x7f7ce801c960 comp rx=0 tx=0).stop 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 -- 172.21.15.153:0/783861416 shutdown_connections 2024-09-10T20:28:48.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 --2- 172.21.15.153:0/783861416 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf809ce10 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 --2- 172.21.15.153:0/783861416 >> v2:172.21.15.153:3300/0 conn(0x7f7cf81364f0 0x7f7cf81368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 -- 172.21.15.153:0/783861416 >> 172.21.15.153:0/783861416 conn(0x7f7cf80a71b0 msgr2=0x7f7cf80a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 -- 172.21.15.153:0/783861416 shutdown_connections 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 -- 172.21.15.153:0/783861416 wait complete. 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient: build_initial_monmap 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient: monmap: 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:48.763 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 librados: starting msgr at 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 librados: starting objecter 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 Processor -- start 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 -- start start 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 librados: setting wanted keys 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 librados: calling monclient init 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient: init 2024-09-10T20:28:48.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf80a1990 0x7f7cf812caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient(hunting): picked mon.noname-b con 0x7f7cf80a1990 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf80717b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient(hunting): picked mon.noname-c con 0x7f7cf81360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf81364f0 0x7f7cf8071cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient(hunting): picked mon.noname-a con 0x7f7cf81364f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf8140e70 con 0x7f7cf81364f0 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7cf80a08a0 con 0x7f7cf80a1990 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7cf8140ff0 con 0x7f7cf81360a0 2024-09-10T20:28:48.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7d004b2640 10 monclient(hunting): authenticate will time out at 1088.894288s 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf80717b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfea4f640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf81364f0 msgr2=0x7f7cf8071cf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfea4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf81364f0 0x7f7cf8071cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf80717b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37160/0 (socket says 172.21.15.153:37160) 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 1 -- 172.21.15.153:0/1412120443 learned_addr learned my addr 172.21.15.153:0/1412120443 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient(hunting): get_auth_request con 0x7f7cf81360a0 auth_method 0 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:48.766 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfe24e640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf80a1990 0x7f7cf812caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient(hunting): handle_auth_done global_id 24224 payload 824 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 1 -- 172.21.15.153:0/1412120443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf80a1990 msgr2=0x7f7cf812caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf80a1990 0x7f7cf812caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 1 -- 172.21.15.153:0/1412120443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf81364f0 msgr2=0x7f7cf8071cf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf81364f0 0x7f7cf8071cf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient: _finish_hunting 0 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 1 monclient: found mon.noname-c 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfda4d640 1 -- 172.21.15.153:0/1412120443 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cf8073230 con 0x7f7cf81360a0 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfe24e640 10 monclient: get_auth_request con 0x7f7cf80a1990 auth_method 0 2024-09-10T20:28:48.767 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.750+0000 7f7cfe24e640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf80a1990 0x7f7cf812caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:28:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cfda4d640 1 -- 172.21.15.153:0/1412120443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 msgr2=0x7f7cf80717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cfda4d640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf80717b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7ce801c790 tx=0x7f7ce8019700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 client.?.objecter ms_handle_connect 0x7f7cf81360a0 2024-09-10T20:28:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:48.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cfda4d640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf80717b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7ce801c790 tx=0x7f7ce8019700 comp rx=0 tx=0).queue_received will delay after 1.726e+09 on 0x7f7ce80b8880 config(0 keys) 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 1 -- 172.21.15.153:0/1412120443 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80bc020 con 0x7f7cf81360a0 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 monclient: dump: 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.771 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 monclient: _finish_auth 0 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 1 -- 172.21.15.153:0/1412120443 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ce80b8880 con 0x7f7cf81360a0 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 5 monclient: authenticate success, global_id 24224 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 10 monclient: _renew_subs 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cf80a6970 con 0x7f7cf81360a0 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf7fff640 4 set_mon_vals no callback set 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 10 client.24224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 10 monclient: _renew_subs 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:48.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cf8137600 con 0x7f7cf81360a0 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 1 librados: init done 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf81368d0 con 0x7f7cf81360a0 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 1 -- 172.21.15.153:0/1412120443 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80b8c50 con 0x7f7cf81360a0 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 10 monclient: dump: 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:48.773 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 1 -- 172.21.15.153:0/1412120443 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f7ce80bf020 con 0x7f7cf81360a0 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cf6ffd640 1 --2- 172.21.15.153:0/1412120443 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7cd803ce50 0x7f7cd803f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cfe24e640 1 --2- 172.21.15.153:0/1412120443 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7cd803ce50 0x7f7cd803f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.754+0000 7f7cfe24e640 10 monclient: get_auth_request con 0x7f7cd803ce50 auth_method 0 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.758+0000 7f7cfe24e640 1 --2- 172.21.15.153:0/1412120443 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7cd803ce50 0x7f7cd803f310 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0003990 tx=0x7f7ce0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.758+0000 7f7cf6ffd640 10 client.24224.objecter ms_handle_connect 0x7f7cd803ce50 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.758+0000 7f7cf6ffd640 1 -- 172.21.15.153:0/1412120443 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(104..104 src has 1..104) ==== 6775+0+0 (secure 0 0 0) 0x7f7ce80eb070 con 0x7f7cf81360a0 2024-09-10T20:28:48.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.758+0000 7f7cf6ffd640 10 client.24224.objecter ms_dispatch 0x7f7cf8000ba0 osd_map(104..104 src has 1..104) 2024-09-10T20:28:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.758+0000 7f7cf6ffd640 3 client.24224.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T20:28:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.758+0000 7f7cf6ffd640 3 client.24224.objecter handle_osd_map decoding full epoch 104 2024-09-10T20:28:48.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:48.758+0000 7f7cf6ffd640 20 client.24224.objecter dump_active .. 0 homeless 2024-09-10T20:28:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.382+0000 7f7cf6ffd640 1 -- 172.21.15.153:0/1412120443 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f7ce80165e0 con 0x7f7cf81360a0 2024-09-10T20:28:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.382+0000 7f7cf6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.382+0000 7f7cf6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:49.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.610+0000 7f7d004b2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:28:49.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.610+0000 7f7d004b2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:49.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.610+0000 7f7d004b2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:49.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.610+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7cf8137100 con 0x7f7cf81360a0 2024-09-10T20:28:49.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7cf6ffd640 1 -- 172.21.15.153:0/1412120443 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f7ce80b8df0 con 0x7f7cf81360a0 2024-09-10T20:28:49.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7cf6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:49.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7cf6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:49.620 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:28:49.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 10 librados: watch_flush enter 2024-09-10T20:28:49.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 10 librados: watch_flush exit 2024-09-10T20:28:49.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 20 client.24224.objecter shutdown clearing up homeless session... 2024-09-10T20:28:49.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 10 client.24224.objecter successfully canceled tick 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7cd803ce50 msgr2=0x7f7cd803f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 --2- 172.21.15.153:0/1412120443 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7cd803ce50 0x7f7cd803f310 secure :-1 s=READY pgs=3258 cs=0 l=1 rev1=1 crypto rx=0x7f7ce0003990 tx=0x7f7ce0008040 comp rx=0 tx=0).stop 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 10 monclient: shutdown 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 msgr2=0x7f7cf80717b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf80717b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f7ce801c790 tx=0x7f7ce8019700 comp rx=0 tx=0).stop 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 shutdown_connections 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cf81364f0 0x7f7cf8071cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 --2- 172.21.15.153:0/1412120443 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7cd803ce50 0x7f7cd803f310 unknown :-1 s=CLOSED pgs=3258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cf81360a0 0x7f7cf80717b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 --2- 172.21.15.153:0/1412120443 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cf80a1990 0x7f7cf812caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:49.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 >> 172.21.15.153:0/1412120443 conn(0x7f7cf80a71b0 msgr2=0x7f7cf80a61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:49.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 shutdown_connections 2024-09-10T20:28:49.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 -- 172.21.15.153:0/1412120443 wait complete. 2024-09-10T20:28:49.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:49.614+0000 7f7d004b2640 1 librados: shutdown 2024-09-10T20:28:52.638 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:28:52.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient: get_monmap_and_config 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient: build_initial_monmap 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient: monmap: 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:52.766807+0000 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:52.766807+0000 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:52.769 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:52.770 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:52.770 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 1 Processor -- start 2024-09-10T20:28:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 1 -- start start 2024-09-10T20:28:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient: init 2024-09-10T20:28:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient: _add_conns ranks=[1,3,4,2,0] 2024-09-10T20:28:52.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f20a01364f0 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f20a01360a0 0x7f20a013b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient(hunting): picked mon.noname-a con 0x7f20a01360a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f20a00a1990 0x7f20a013d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient(hunting): picked mon.noname-c con 0x7f20a00a1990 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20a00a27a0 con 0x7f20a01364f0 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f20a00a2550 con 0x7f20a01360a0 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f20a00a2a80 con 0x7f20a00a1990 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a68bd640 10 monclient(hunting): authenticate will time out at 1092.906540s 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f209f7fe640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f20a01360a0 msgr2=0x7f20a013b420 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:28:52.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f209f7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f20a01360a0 0x7f20a013b420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f20a00a1990 0x7f20a013d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f20a00a1990 0x7f20a013d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.176:3300/0 says I am v2:172.21.15.153:50634/0 (socket says 172.21.15.153:50634) 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 1 -- 172.21.15.153:0/2790062195 learned_addr learned my addr 172.21.15.153:0/2790062195 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 10 monclient(hunting): get_auth_request con 0x7f20a00a1990 auth_method 0 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:52.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.762+0000 7f20a4e5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f209ffff640 1 -- 172.21.15.153:0/2790062195 >> v1:172.21.15.176:6789/0 conn(0x7f20a01364f0 legacy=0x7f20a01368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f20a4e5a640 10 monclient(hunting): handle_auth_done global_id 19699 payload 274 2024-09-10T20:28:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f20a4e5a640 1 -- 172.21.15.153:0/2790062195 >> v2:172.21.15.153:3300/0 conn(0x7f20a01360a0 msgr2=0x7f20a013b420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:28:52.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f20a4e5a640 1 --2- 172.21.15.153:0/2790062195 >> v2:172.21.15.153:3300/0 conn(0x7f20a01360a0 0x7f20a013b420 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f20a4e5a640 1 -- 172.21.15.153:0/2790062195 >> v1:172.21.15.176:6789/0 conn(0x7f20a01364f0 legacy=0x7f20a01368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f20a4e5a640 10 monclient: _finish_hunting 0 2024-09-10T20:28:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f20a4e5a640 1 monclient: found mon.noname-c 2024-09-10T20:28:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f20a4e5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f20a4e5a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:28:52.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.766+0000 7f20a4e5a640 1 -- 172.21.15.153:0/2790062195 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a00a2d60 con 0x7f20a00a1990 2024-09-10T20:28:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f20a4e5a640 1 -- 172.21.15.153:0/2790062195 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 msgr2=0x7f20a013d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:52.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f20a4e5a640 1 --2- 172.21.15.153:0/2790062195 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 0x7f20a013d970 secure :-1 s=READY pgs=5492 cs=0 l=1 rev1=1 crypto rx=0x7f20a01370d0 tx=0x7f209801c480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=167ab34ac9378d64 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 1 -- 172.21.15.153:0/2790062195 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20980aa6e0 con 0x7f20a00a1990 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 10 monclient: dump: 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:52.780 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 10 monclient: _finish_auth 0 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 10 monclient: _check_auth_tickets 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 1 -- 172.21.15.153:0/2790062195 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20980af070 con 0x7f20a00a1990 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f20a68bd640 5 monclient: authenticate success, global_id 19699 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f20a68bd640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 1 -- 172.21.15.153:0/2790062195 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20980aaaa0 con 0x7f20a00a1990 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f20a68bd640 4 set_mon_vals no callback set 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:28:52.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.770+0000 7f209effd640 10 monclient: dump: 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient: shutdown 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 -- 172.21.15.153:0/2790062195 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 msgr2=0x7f20a013d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 --2- 172.21.15.153:0/2790062195 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 0x7f20a013d970 secure :-1 s=READY pgs=5492 cs=0 l=1 rev1=1 crypto rx=0x7f20a01370d0 tx=0x7f209801c480 comp rx=0 tx=0).stop 2024-09-10T20:28:52.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 -- 172.21.15.153:0/2790062195 shutdown_connections 2024-09-10T20:28:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 --2- 172.21.15.153:0/2790062195 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 0x7f20a013d970 unknown :-1 s=CLOSED pgs=5492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 --2- 172.21.15.153:0/2790062195 >> v2:172.21.15.153:3300/0 conn(0x7f20a01360a0 0x7f20a013b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 -- 172.21.15.153:0/2790062195 >> 172.21.15.153:0/2790062195 conn(0x7f20a00a71b0 msgr2=0x7f20a00a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 -- 172.21.15.153:0/2790062195 shutdown_connections 2024-09-10T20:28:52.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 -- 172.21.15.153:0/2790062195 wait complete. 2024-09-10T20:28:52.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient: build_initial_monmap 2024-09-10T20:28:52.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient: monmap: 2024-09-10T20:28:52.786 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:52.786 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:52.786 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 librados: starting msgr at 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 librados: starting objecter 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 Processor -- start 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 -- start start 2024-09-10T20:28:52.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 librados: setting wanted keys 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 librados: calling monclient init 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient: init 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 0x7f20a012f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient(hunting): picked mon.noname-b con 0x7f20a00a1990 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20a01360a0 0x7f20a012fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient(hunting): picked mon.noname-a con 0x7f20a01360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20a01364f0 0x7f20a01d0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient(hunting): picked mon.noname-c con 0x7f20a01364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f20a0140af0 con 0x7f20a01360a0 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:52.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f20a0140970 con 0x7f20a00a1990 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f20a0140c70 con 0x7f20a01364f0 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a68bd640 10 monclient(hunting): authenticate will time out at 1092.917774s 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f209f7fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20a01360a0 msgr2=0x7f20a012fe40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f209f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20a01360a0 0x7f20a012fe40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20a01364f0 0x7f20a01d0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20a01364f0 0x7f20a01d0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59212/0 (socket says 172.21.15.153:59212) 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 1 -- 172.21.15.153:0/1042599192 learned_addr learned my addr 172.21.15.153:0/1042599192 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient(hunting): get_auth_request con 0x7f20a01364f0 auth_method 0 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:52.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f209ffff640 1 --2- 172.21.15.153:0/1042599192 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 0x7f20a012f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f209ffff640 10 monclient(hunting): get_auth_request con 0x7f20a00a1990 auth_method 0 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f209ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f209ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f209ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient(hunting): handle_auth_done global_id 24227 payload 824 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 1 -- 172.21.15.153:0/1042599192 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 msgr2=0x7f20a012f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 1 --2- 172.21.15.153:0/1042599192 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 0x7f20a012f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:52.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 1 -- 172.21.15.153:0/1042599192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20a01360a0 msgr2=0x7f20a012fe40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 1 --2- 172.21.15.153:0/1042599192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20a01360a0 0x7f20a012fe40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient: _finish_hunting 0 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 1 monclient: found mon.noname-c 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.774+0000 7f20a4e5a640 1 -- 172.21.15.153:0/1042599192 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20a009e0a0 con 0x7f20a01364f0 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a4e5a640 1 -- 172.21.15.153:0/1042599192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20a01364f0 msgr2=0x7f20a01d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a4e5a640 1 --2- 172.21.15.153:0/1042599192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20a01364f0 0x7f20a01d0200 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f20980ac040 tx=0x7f209801acc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 1 -- 172.21.15.153:0/1042599192 DispatchQueue::entry inject delay of 0.002000 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 client.?.objecter ms_handle_connect 0x7f20a01364f0 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 1 -- 172.21.15.153:0/1042599192 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20980b3a10 con 0x7f20a01364f0 2024-09-10T20:28:52.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 monclient: dump: 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:52.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 monclient: _finish_auth 0 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 monclient: _check_auth_tickets 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 1 -- 172.21.15.153:0/1042599192 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20980b3bb0 con 0x7f20a01364f0 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 5 monclient: authenticate success, global_id 24227 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 1 -- 172.21.15.153:0/1042599192 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20980bd3f0 con 0x7f20a01364f0 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 10 monclient: _renew_subs 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20a013cab0 con 0x7f20a01364f0 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209e7fc640 4 set_mon_vals no callback set 2024-09-10T20:28:52.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 10 monclient: dump: 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:52.794 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:52.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 10 client.24227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:52.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 10 monclient: _renew_subs 2024-09-10T20:28:52.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:52.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f20a00a6840 con 0x7f20a01364f0 2024-09-10T20:28:52.795 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 1 librados: init done 2024-09-10T20:28:52.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:52.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:52.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:52.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 1 -- 172.21.15.153:0/1042599192 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f20980a9070 con 0x7f20a01364f0 2024-09-10T20:28:52.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20a01368d0 con 0x7f20a01364f0 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.778+0000 7f209cff9640 1 --2- 172.21.15.153:0/1042599192 >> v2:172.21.15.176:6812/3003473398 conn(0x7f208003cea0 0x7f208003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.782+0000 7f209cff9640 1 -- 172.21.15.153:0/1042599192 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(104..104 src has 1..104) ==== 6775+0+0 (secure 0 0 0) 0x7f20980ea7e0 con 0x7f20a01364f0 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.782+0000 7f209cff9640 10 client.24227.objecter ms_dispatch 0x7f20a0000ba0 osd_map(104..104 src has 1..104) 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.782+0000 7f209cff9640 3 client.24227.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.782+0000 7f209cff9640 3 client.24227.objecter handle_osd_map decoding full epoch 104 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.782+0000 7f209cff9640 20 client.24227.objecter dump_active .. 0 homeless 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.782+0000 7f209ffff640 1 --2- 172.21.15.153:0/1042599192 >> v2:172.21.15.176:6812/3003473398 conn(0x7f208003cea0 0x7f208003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.782+0000 7f209ffff640 10 monclient: get_auth_request con 0x7f208003cea0 auth_method 0 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.782+0000 7f209cff9640 10 client.24227.objecter ms_handle_connect 0x7f208003cea0 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.782+0000 7f209ffff640 1 --2- 172.21.15.153:0/1042599192 >> v2:172.21.15.176:6812/3003473398 conn(0x7f208003cea0 0x7f208003f360 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f20900012f0 tx=0x7f2090009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.786+0000 7f209cff9640 1 -- 172.21.15.153:0/1042599192 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f20a009e0a0 con 0x7f20a01364f0 2024-09-10T20:28:52.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.786+0000 7f209cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:52.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.786+0000 7f209cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.998+0000 7f20a68bd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:28:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.998+0000 7f20a68bd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.998+0000 7f20a68bd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:52.998+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f20a0130380 con 0x7f20a01364f0 2024-09-10T20:28:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f209cff9640 1 -- 172.21.15.153:0/1042599192 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f2098016190 con 0x7f20a01364f0 2024-09-10T20:28:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f209cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f209cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:53.009 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":10,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:28:53.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 10 librados: watch_flush enter 2024-09-10T20:28:53.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 10 librados: watch_flush exit 2024-09-10T20:28:53.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 20 client.24227.objecter shutdown clearing up homeless session... 2024-09-10T20:28:53.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 10 client.24227.objecter successfully canceled tick 2024-09-10T20:28:53.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 >> v2:172.21.15.176:6812/3003473398 conn(0x7f208003cea0 msgr2=0x7f208003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:53.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 --2- 172.21.15.153:0/1042599192 >> v2:172.21.15.176:6812/3003473398 conn(0x7f208003cea0 0x7f208003f360 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7f20900012f0 tx=0x7f2090009040 comp rx=0 tx=0).stop 2024-09-10T20:28:53.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 10 monclient: shutdown 2024-09-10T20:28:53.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:53.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20a01364f0 msgr2=0x7f20a01d0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 --2- 172.21.15.153:0/1042599192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20a01364f0 0x7f20a01d0200 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f20980ac040 tx=0x7f209801acc0 comp rx=0 tx=0).stop 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 shutdown_connections 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 --2- 172.21.15.153:0/1042599192 >> v2:172.21.15.176:6812/3003473398 conn(0x7f208003cea0 0x7f208003f360 unknown :-1 s=CLOSED pgs=3259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 --2- 172.21.15.153:0/1042599192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f20a01364f0 0x7f20a01d0200 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 --2- 172.21.15.153:0/1042599192 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f20a01360a0 0x7f20a012fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 --2- 172.21.15.153:0/1042599192 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f20a00a1990 0x7f20a012f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 >> 172.21.15.153:0/1042599192 conn(0x7f20a00a71b0 msgr2=0x7f20a0132a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 shutdown_connections 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 -- 172.21.15.153:0/1042599192 wait complete. 2024-09-10T20:28:53.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:53.002+0000 7f20a68bd640 1 librados: shutdown 2024-09-10T20:28:56.027 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:28:56.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient: get_monmap_and_config 2024-09-10T20:28:56.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient: build_initial_monmap 2024-09-10T20:28:56.154 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient: monmap: 2024-09-10T20:28:56.154 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:56.154 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:56.154 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:56.152077+0000 2024-09-10T20:28:56.154 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:56.152077+0000 2024-09-10T20:28:56.155 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:56.155 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:56.155 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:56.155 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:56.155 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:56.155 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:56.155 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:56.155 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 1 Processor -- start 2024-09-10T20:28:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 1 -- start start 2024-09-10T20:28:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient: init 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient: _add_conns ranks=[4,2,1,3,0] 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6ac01364f0 0x7f6ac01368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac01364f0 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac009ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac01360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6ac00a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ac00a27a0 con 0x7f6ac00a1990 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f6ac00a2a80 con 0x7f6ac01364f0 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac00a2d60 con 0x7f6ac01360a0 2024-09-10T20:28:56.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6ac65fd640 10 monclient(hunting): authenticate will time out at 1096.291764s 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.146+0000 7f6abf7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac009ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac009ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59216/0 (socket says 172.21.15.153:59216) 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 1 -- 172.21.15.153:0/2892499128 learned_addr learned my addr 172.21.15.153:0/2892499128 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient(hunting): get_auth_request con 0x7f6ac01360a0 auth_method 0 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abffff640 1 --2- 172.21.15.153:0/2892499128 >> v2:172.21.15.176:3300/0 conn(0x7f6ac01364f0 0x7f6ac01368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abffff640 10 monclient(hunting): get_auth_request con 0x7f6ac01364f0 auth_method 0 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient(hunting): handle_auth_done global_id 24230 payload 274 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 1 -- 172.21.15.153:0/2892499128 >> v2:172.21.15.176:3300/0 conn(0x7f6ac01364f0 msgr2=0x7f6ac01368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 1 --2- 172.21.15.153:0/2892499128 >> v2:172.21.15.176:3300/0 conn(0x7f6ac01364f0 0x7f6ac01368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 1 -- 172.21.15.153:0/2892499128 >> v1:172.21.15.176:6789/0 conn(0x7f6ac00a1990 legacy=0x7f6ac009d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 1 monclient: found mon.noname-b 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:28:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.150+0000 7f6abf7fe640 1 -- 172.21.15.153:0/2892499128 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac0136e10 con 0x7f6ac01360a0 2024-09-10T20:28:56.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 -- 172.21.15.153:0/2892499128 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 msgr2=0x7f6ac009ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:56.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 --2- 172.21.15.153:0/2892499128 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac009ce10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00a12f0 tx=0x7f6aa801c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9237c1ff4c6633e2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:56.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 1 -- 172.21.15.153:0/2892499128 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80b0070 con 0x7f6ac01360a0 2024-09-10T20:28:56.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:56.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:28:56.167 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 10 monclient: dump: 2024-09-10T20:28:56.167 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:56.167 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:56.167 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 10 monclient: _finish_auth 0 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 10 monclient: _check_auth_tickets 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 1 -- 172.21.15.153:0/2892499128 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa80ab600 con 0x7f6ac01360a0 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:56.168 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 1 -- 172.21.15.153:0/2892499128 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80ab9d0 con 0x7f6ac01360a0 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 5 monclient: authenticate success, global_id 24230 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient: get_monmap_and_config success 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 4 set_mon_vals no callback set 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abeffd640 10 monclient: dump: 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient: shutdown 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 -- 172.21.15.153:0/2892499128 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 msgr2=0x7f6ac009ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 --2- 172.21.15.153:0/2892499128 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac009ce10 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6ac00a12f0 tx=0x7f6aa801c470 comp rx=0 tx=0).stop 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 -- 172.21.15.153:0/2892499128 shutdown_connections 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 --2- 172.21.15.153:0/2892499128 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac009ce10 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 --2- 172.21.15.153:0/2892499128 >> v2:172.21.15.176:3300/0 conn(0x7f6ac01364f0 0x7f6ac01368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 -- 172.21.15.153:0/2892499128 >> 172.21.15.153:0/2892499128 conn(0x7f6ac00a71b0 msgr2=0x7f6ac00a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 -- 172.21.15.153:0/2892499128 shutdown_connections 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 -- 172.21.15.153:0/2892499128 wait complete. 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient: build_initial_monmap 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient: monmap: 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 librados: starting msgr at 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 librados: starting objecter 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 Processor -- start 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 -- start start 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 librados: setting wanted keys 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 librados: calling monclient init 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient: init 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:28:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac00a1990 0x7f6ac012f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient(hunting): picked mon.noname-a con 0x7f6ac00a1990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac0073d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient(hunting): picked mon.noname-c con 0x7f6ac01360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac01364f0 0x7f6ac00742c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient(hunting): picked mon.noname-b con 0x7f6ac01364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac0140ff0 con 0x7f6ac00a1990 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6ac0140e70 con 0x7f6ac01364f0 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6ac0141170 con 0x7f6ac01360a0 2024-09-10T20:28:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac65fd640 10 monclient(hunting): authenticate will time out at 1096.298899s 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abffff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac00a1990 msgr2=0x7f6ac012f930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abffff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac00a1990 0x7f6ac012f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac0073d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac0073d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59222/0 (socket says 172.21.15.153:59222) 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 -- 172.21.15.153:0/632383880 learned_addr learned my addr 172.21.15.153:0/632383880 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient(hunting): get_auth_request con 0x7f6ac01360a0 auth_method 0 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:28:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac4b9a640 1 --2- 172.21.15.153:0/632383880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac01364f0 0x7f6ac00742c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient(hunting): handle_auth_done global_id 24233 payload 824 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 -- 172.21.15.153:0/632383880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac01364f0 msgr2=0x7f6ac00742c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 --2- 172.21.15.153:0/632383880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac01364f0 0x7f6ac00742c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 -- 172.21.15.153:0/632383880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac00a1990 msgr2=0x7f6ac012f930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 --2- 172.21.15.153:0/632383880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac00a1990 0x7f6ac012f930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 monclient: found mon.noname-c 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6abf7fe640 1 -- 172.21.15.153:0/632383880 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ac01ceb30 con 0x7f6ac01360a0 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac4b9a640 10 monclient: get_auth_request con 0x7f6ac01364f0 auth_method 0 2024-09-10T20:28:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.154+0000 7f6ac4b9a640 1 --2- 172.21.15.153:0/632383880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac01364f0 0x7f6ac00742c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abf7fe640 1 -- 172.21.15.153:0/632383880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 msgr2=0x7f6ac0073d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abf7fe640 1 --2- 172.21.15.153:0/632383880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac0073d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80ae040 tx=0x7f6aa801acb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 client.?.objecter ms_handle_connect 0x7f6ac01360a0 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 1 -- 172.21.15.153:0/632383880 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80ba020 con 0x7f6ac01360a0 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 monclient: dump: 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 monclient: _finish_auth 0 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 monclient: _check_auth_tickets 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 1 -- 172.21.15.153:0/632383880 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6aa80b6850 con 0x7f6ac01360a0 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 5 monclient: authenticate success, global_id 24233 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 10 monclient: _renew_subs 2024-09-10T20:28:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ac00a69a0 con 0x7f6ac01360a0 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 1 -- 172.21.15.153:0/632383880 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6aa80b0040 con 0x7f6ac01360a0 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 monclient: dump: 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:28:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abe7fc640 4 set_mon_vals no callback set 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 10 client.24233.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 10 monclient: _renew_subs 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ac009e0a0 con 0x7f6ac01360a0 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 1 librados: init done 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ac0071fb0 con 0x7f6ac01360a0 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 1 -- 172.21.15.153:0/632383880 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f6aa80bd020 con 0x7f6ac01360a0 2024-09-10T20:28:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 1 --2- 172.21.15.153:0/632383880 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6aa003cea0 0x7f6aa003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 1 -- 172.21.15.153:0/632383880 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(104..104 src has 1..104) ==== 6775+0+0 (secure 0 0 0) 0x7f6aa80ea000 con 0x7f6ac01360a0 2024-09-10T20:28:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 10 client.24233.objecter ms_dispatch 0x7f6ac0000ba0 osd_map(104..104 src has 1..104) 2024-09-10T20:28:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 3 client.24233.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T20:28:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 3 client.24233.objecter handle_osd_map decoding full epoch 104 2024-09-10T20:28:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abcff9640 20 client.24233.objecter dump_active .. 0 homeless 2024-09-10T20:28:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abffff640 1 --2- 172.21.15.153:0/632383880 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6aa003cea0 0x7f6aa003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.158+0000 7f6abffff640 10 monclient: get_auth_request con 0x7f6aa003cea0 auth_method 0 2024-09-10T20:28:56.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.162+0000 7f6abffff640 1 --2- 172.21.15.153:0/632383880 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6aa003cea0 0x7f6aa003f360 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009660 tx=0x7f6ab0009930 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:28:56.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.162+0000 7f6abcff9640 10 client.24233.objecter ms_handle_connect 0x7f6aa003cea0 2024-09-10T20:28:56.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.162+0000 7f6abcff9640 1 -- 172.21.15.153:0/632383880 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f6aa80c4070 con 0x7f6ac01360a0 2024-09-10T20:28:56.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.162+0000 7f6abcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:28:56.181 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.162+0000 7f6abcff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:28:56.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.390+0000 7f6ac65fd640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:28:56.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.390+0000 7f6ac65fd640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:56.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.390+0000 7f6ac65fd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:28:56.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.390+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6ac01368d0 con 0x7f6ac01360a0 2024-09-10T20:28:56.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6abcff9640 1 -- 172.21.15.153:0/632383880 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7f6ac00a69a0 con 0x7f6ac01360a0 2024-09-10T20:28:56.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6abcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:28:56.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6abcff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:28:56.400 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":18,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 10 librados: watch_flush enter 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 10 librados: watch_flush exit 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 20 client.24233.objecter shutdown clearing up homeless session... 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 10 client.24233.objecter successfully canceled tick 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6aa003cea0 msgr2=0x7f6aa003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 --2- 172.21.15.153:0/632383880 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6aa003cea0 0x7f6aa003f360 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7f6ab0009660 tx=0x7f6ab0009930 comp rx=0 tx=0).stop 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 10 monclient: shutdown 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 msgr2=0x7f6ac0073d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:28:56.402 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 --2- 172.21.15.153:0/632383880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac0073d80 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6aa80ae040 tx=0x7f6aa801acb0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 shutdown_connections 2024-09-10T20:28:56.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 --2- 172.21.15.153:0/632383880 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6ac01364f0 0x7f6ac00742c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 --2- 172.21.15.153:0/632383880 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6aa003cea0 0x7f6aa003f360 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 --2- 172.21.15.153:0/632383880 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6ac01360a0 0x7f6ac0073d80 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 --2- 172.21.15.153:0/632383880 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6ac00a1990 0x7f6ac012f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:28:56.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 >> 172.21.15.153:0/632383880 conn(0x7f6ac00a71b0 msgr2=0x7f6ac012c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:28:56.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 shutdown_connections 2024-09-10T20:28:56.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 -- 172.21.15.153:0/632383880 wait complete. 2024-09-10T20:28:56.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:56.394+0000 7f6ac65fd640 1 librados: shutdown 2024-09-10T20:28:59.418 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:28:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient: get_monmap_and_config 2024-09-10T20:28:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient: build_initial_monmap 2024-09-10T20:28:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient: monmap: 2024-09-10T20:28:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:28:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:28:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:28:59.547368+0000 2024-09-10T20:28:59.549 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:28:59.547368+0000 2024-09-10T20:28:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:28:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:28:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:28:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:28:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:28:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:28:59.550 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:28:59.550 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:28:59.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 1 Processor -- start 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 1 -- start start 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient: init 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient: _reopen_session rank -1 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4f7c0a5ab0 addr v1:172.21.15.153:6789/0 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7c0a65a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c13e1b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c070780 con 0x7f4f7c0a5ab0 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4f7c0a3010 con 0x7f4f7c0a65a0 2024-09-10T20:28:59.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:28:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c0709c0 con 0x7f4f7c13e1b0 2024-09-10T20:28:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient(hunting): _renew_subs 2024-09-10T20:28:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f830a0640 10 monclient(hunting): authenticate will time out at 1099.687059s 2024-09-10T20:28:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f80e3c640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f4f7c0a5ab0 legacy=0x7f4f7c0a5e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-10T20:28:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:59238/0 (socket says 172.21.15.153:59238) 2024-09-10T20:28:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 learned_addr learned my addr 172.21.15.153:0/2156337481 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:28:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 10 monclient(hunting): get_auth_request con 0x7f4f7c13e1b0 auth_method 0 2024-09-10T20:28:59.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:59.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:59.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:59.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 msgr2=0x7f4f7c1405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:28:59.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 msgr2=0x7f4f7c1405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f7c0a65a0 auth_method 0 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 msgr2=0x7f4f7c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 msgr2=0x7f4f7c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:59.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.541+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:28:59.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f80e3c640 1 -- 172.21.15.153:0/2156337481 >> v1:172.21.15.153:6789/0 conn(0x7f4f7c0a5ab0 legacy=0x7f4f7c0a5e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-10T20:28:59.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f8163d640 10 monclient(hunting): get_auth_request con 0x7f4f7c13e1b0 auth_method 0 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f8163d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f8163d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f8163d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 msgr2=0x7f4f7c1405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 msgr2=0x7f4f7c1405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f7c0a65a0 auth_method 0 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:28:59.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:28:59.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f7bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:28:59.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 msgr2=0x7f4f7c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:28:59.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 msgr2=0x7f4f7c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:28:59.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:28:59.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:28:59.745+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:29:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f8163d640 10 monclient(hunting): get_auth_request con 0x7f4f7c13e1b0 auth_method 0 2024-09-10T20:29:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f8163d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f8163d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f8163d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f80e3c640 1 -- 172.21.15.153:0/2156337481 >> v1:172.21.15.153:6789/0 conn(0x7f4f7c0a5ab0 legacy=0x7f4f7c0a5e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-10T20:29:00.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 msgr2=0x7f4f7c1405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 msgr2=0x7f4f7c1405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f7c0a65a0 auth_method 0 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f7bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 msgr2=0x7f4f7c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 msgr2=0x7f4f7c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:00.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:00.159 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.149+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:29:00.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f80e3c640 1 -- 172.21.15.153:0/2156337481 >> v1:172.21.15.153:6789/0 conn(0x7f4f7c0a5ab0 legacy=0x7f4f7c0a5e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-10T20:29:00.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f7c0a65a0 auth_method 0 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f7bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 msgr2=0x7f4f7c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 msgr2=0x7f4f7c13dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f8163d640 10 monclient(hunting): get_auth_request con 0x7f4f7c13e1b0 auth_method 0 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f8163d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f8163d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:00.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f8163d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:00.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:29:00.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 msgr2=0x7f4f7c1405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:29:00.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 msgr2=0x7f4f7c1405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:00.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:00.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:00.953+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:29:02.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 10 monclient(hunting): tick 2024-09-10T20:29:02.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-10T20:29:02.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 1 monclient(hunting): continuing hunt 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 msgr2=0x7f4f7c1405a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c13e1b0 0x7f4f7c1405a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 msgr2=0x7f4f7c13dc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f7c0a65a0 0x7f4f7c13dc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 >> v1:172.21.15.153:6789/0 conn(0x7f4f7c0a5ab0 legacy=0x7f4f7c0a5e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 10 monclient: _add_conns ranks=[3,1] 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 0x7f4f68005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 10 monclient(hunting): picked mon.noname-a con 0x7f4f680036b0 addr v2:172.21.15.153:3300/0 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f68006200 addr v1:172.21.15.176:6789/0 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c0a26e0 con 0x7f4f68006200 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:02.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f4f7c0a3010 con 0x7f4f680036b0 2024-09-10T20:29:02.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f7a7fc640 10 monclient(hunting): _renew_subs 2024-09-10T20:29:02.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f80e3c640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:02.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.541+0000 7f4f80e3c640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:02.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.545+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v1:172.21.15.176:6789/0 conn(0x7f4f68006200 legacy=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:02.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.745+0000 7f4f80e3c640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:02.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:02.745+0000 7f4f80e3c640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:29:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:03.149+0000 7f4f80e3c640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:03.153 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:03.149+0000 7f4f80e3c640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:29:03.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:03.953+0000 7f4f80e3c640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:03.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:03.953+0000 7f4f80e3c640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:29:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient(hunting): tick 2024-09-10T20:29:05.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient(hunting): _check_auth_tickets 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 1 monclient(hunting): continuing hunt 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.153:3300/0 conn(0x7f4f680036b0 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 >> v1:172.21.15.176:6789/0 conn(0x7f4f68006200 legacy=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient: _add_conns ranks=[1,4,2,3,0] 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 reap_dead start 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4f6800a0f0 addr v1:172.21.15.176:6789/0 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient(hunting): picked mon.noname-c con 0x7f4f68008e90 addr v2:172.21.15.176:3300/0 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient(hunting): picked mon.noname-b con 0x7f4f68009270 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:29:05.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f7c070780 con 0x7f4f6800a0f0 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f4f7c0a3010 con 0x7f4f68008e90 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 1 -- 172.21.15.153:0/2156337481 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c0709c0 con 0x7f4f68009270 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.541+0000 7f4f7a7fc640 10 monclient(hunting): _renew_subs 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f68008e90 auth_method 0 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f8163d640 10 monclient(hunting): get_auth_request con 0x7f4f68009270 auth_method 0 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f8163d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:05.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f8163d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f8163d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:05.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.545+0000 7f4f80e3c640 1 -- 172.21.15.153:0/2156337481 >> v1:172.21.15.176:6789/0 conn(0x7f4f6800a0f0 legacy=0x7f4f6800a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:05.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:05.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f8163d640 10 monclient(hunting): get_auth_request con 0x7f4f68009270 auth_method 0 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f8163d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f8163d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f8163d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f68008e90 auth_method 0 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f7bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:05.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:05.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:29:05.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:05.749+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:29:06.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:06.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f8163d640 10 monclient(hunting): get_auth_request con 0x7f4f68009270 auth_method 0 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f8163d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f8163d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f8163d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f68008e90 auth_method 0 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f7bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:06.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:29:06.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:06.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.153+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:29:06.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:06.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f8163d640 10 monclient(hunting): get_auth_request con 0x7f4f68009270 auth_method 0 2024-09-10T20:29:06.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f8163d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:06.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f8163d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:06.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f8163d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:06.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request con 0x7f4f68008e90 auth_method 0 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f7bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f7bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f7bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:29:06.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:06.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:06.957+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:29:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.513+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 100659507 0 0) 0x7f4f7c070780 con 0x7f4f6800a0f0 2024-09-10T20:29:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.513+0000 7f4f7b7fe640 10 monclient(hunting): _negotiate 2024-09-10T20:29:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.513+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.513+0000 7f4f7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.513+0000 7f4f7b7fe640 10 monclient(hunting): my global_id is 19711 2024-09-10T20:29:08.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.513+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f64003530 con 0x7f4f6800a0f0 2024-09-10T20:29:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2736010764 0 0) 0x7f4f64003530 con 0x7f4f6800a0f0 2024-09-10T20:29:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f680086c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f680086c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 msgr2=0x7f4f68005b10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 --2- 172.21.15.153:0/2156337481 >> v2:172.21.15.176:3300/0 conn(0x7f4f68008e90 0x7f4f68005b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 monclient: found mon.noname-c-legacy 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.176:6789/0 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f68009720 con 0x7f4f6800a0f0 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3765011595 0 0) 0x7f4f70004660 con 0x7f4f6800a0f0 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T20:29:08.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient: dump: 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] but i'm connected to v1:172.21.15.176:6789/0, reconnecting 2024-09-10T20:29:08.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient: _reopen_session rank -1 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 >> v1:172.21.15.176:6789/0 conn(0x7f4f6800a0f0 legacy=0x7f4f6800a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f640039c0 0x7f4f64003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient(hunting): picked mon.b con 0x7f4f640039c0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f64004810 0x7f4f64004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient(hunting): picked mon.a con 0x7f4f64004810 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f640051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient(hunting): picked mon.c con 0x7f4f68009270 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c0a3010 con 0x7f4f64004810 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c0709c0 con 0x7f4f640039c0 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4f6400bb30 con 0x7f4f68009270 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient(hunting): _renew_subs 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7bfff640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f64004810 msgr2=0x7f4f64004c90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f64004810 0x7f4f64004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f80e3c640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f640039c0 0x7f4f64003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f640051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient(hunting): get_auth_request con 0x7f4f68009270 auth_method 0 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f80e3c640 10 monclient(hunting): get_auth_request con 0x7f4f640039c0 auth_method 0 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f80e3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:08.531 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f80e3c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f80e3c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f80e3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f80e3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f80e3c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient(hunting): handle_auth_done global_id 19711 payload 293 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f640039c0 msgr2=0x7f4f64003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f640039c0 0x7f4f64003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f64004810 msgr2=0x7f4f64004c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f64004810 0x7f4f64004c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.532 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient: _finish_hunting 0 2024-09-10T20:29:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 1 monclient: found mon.c 2024-09-10T20:29:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:29:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f64003530 con 0x7f4f68009270 2024-09-10T20:29:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient: _finish_auth 0 2024-09-10T20:29:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 10 monclient: _check_auth_tickets 2024-09-10T20:29:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f8163d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:29:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f830a0640 5 monclient: authenticate success, global_id 19711 2024-09-10T20:29:08.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.517+0000 7f4f830a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:29:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f640051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f8163d640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f640051d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0a5320 tx=0x7f4f7401e730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61ff77b50e1be616 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f740097e0 con 0x7f4f68009270 2024-09-10T20:29:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:29:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 10 monclient: dump: 2024-09-10T20:29:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.535 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f830a0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f740039a0 con 0x7f4f68009270 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 1 -- 172.21.15.153:0/2156337481 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f74003b90 con 0x7f4f68009270 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f830a0640 10 monclient: get_monmap_and_config success 2024-09-10T20:29:08.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f830a0640 4 set_mon_vals no callback set 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.525+0000 7f4f7b7fe640 10 monclient: dump: 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.537 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient: shutdown 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 -- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 msgr2=0x7f4f640051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 --2- 172.21.15.153:0/2156337481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f68009270 0x7f4f640051d0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4f7c0a5320 tx=0x7f4f7401e730 comp rx=0 tx=0).stop 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f8163d640 1 -- 172.21.15.153:0/2156337481 reap_dead start 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 -- 172.21.15.153:0/2156337481 shutdown_connections 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 -- 172.21.15.153:0/2156337481 >> 172.21.15.153:0/2156337481 conn(0x7f4f7c0aba00 msgr2=0x7f4f7c0a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 -- 172.21.15.153:0/2156337481 shutdown_connections 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 -- 172.21.15.153:0/2156337481 wait complete. 2024-09-10T20:29:08.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient: build_initial_monmap 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient: monmap: 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 librados: starting msgr at 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 librados: starting objecter 2024-09-10T20:29:08.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 Processor -- start 2024-09-10T20:29:08.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 -- start start 2024-09-10T20:29:08.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 librados: setting wanted keys 2024-09-10T20:29:08.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 librados: calling monclient init 2024-09-10T20:29:08.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient: init 2024-09-10T20:29:08.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient: _reopen_session rank -1 2024-09-10T20:29:08.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c137e50 0x7f4f7c138230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7c137e50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c121990 0x7f4f7c121df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient(hunting): picked mon.noname-a con 0x7f4f7c121990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f7c122330 0x7f4f7c122770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7c122330 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c0a3960 con 0x7f4f7c121990 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4f7c0a3ae0 con 0x7f4f7c122330 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f7bfff640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c121990 msgr2=0x7f4f7c121df0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:08.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4f7c1391d0 con 0x7f4f7c137e50 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f7bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c121990 0x7f4f7c121df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient(hunting): _renew_subs 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f830a0640 10 monclient(hunting): authenticate will time out at 1108.676907s 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c137e50 0x7f4f7c138230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c137e50 0x7f4f7c138230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50352/0 (socket says 172.21.15.153:50352) 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 -- 172.21.15.153:0/3823750964 learned_addr learned my addr 172.21.15.153:0/3823750964 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient(hunting): get_auth_request con 0x7f4f7c137e50 auth_method 0 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:08.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f8163d640 1 --2- 172.21.15.153:0/3823750964 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f7c122330 0x7f4f7c122770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient(hunting): handle_auth_done global_id 24236 payload 824 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 -- 172.21.15.153:0/3823750964 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f7c122330 msgr2=0x7f4f7c122770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 --2- 172.21.15.153:0/3823750964 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f7c122330 0x7f4f7c122770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 -- 172.21.15.153:0/3823750964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c121990 msgr2=0x7f4f7c121df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 --2- 172.21.15.153:0/3823750964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c121990 0x7f4f7c121df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient: _finish_hunting 0 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 monclient: found mon.noname-c 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 -- 172.21.15.153:0/3823750964 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f7c124390 con 0x7f4f7c137e50 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f8163d640 10 monclient: get_auth_request con 0x7f4f7c122330 auth_method 0 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f8163d640 1 --2- 172.21.15.153:0/3823750964 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f7c122330 0x7f4f7c122770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:29:08.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 -- 172.21.15.153:0/3823750964 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c137e50 msgr2=0x7f4f7c138230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f80e3c640 1 --2- 172.21.15.153:0/3823750964 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c137e50 0x7f4f7c138230 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4f70010c30 tx=0x7f4f7000e530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f797fa640 10 client.?.objecter ms_handle_connect 0x7f4f7c137e50 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f797fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f797fa640 1 -- 172.21.15.153:0/3823750964 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f7000ee40 con 0x7f4f7c137e50 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f797fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.533+0000 7f4f797fa640 10 monclient: dump: 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.549 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 10 monclient: _finish_auth 0 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 10 monclient: _check_auth_tickets 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 1 -- 172.21.15.153:0/3823750964 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4f70004930 con 0x7f4f7c137e50 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 1 -- 172.21.15.153:0/3823750964 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f70004d20 con 0x7f4f7c137e50 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 10 monclient: dump: 2024-09-10T20:29:08.550 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 5 monclient: authenticate success, global_id 24236 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f7a7fc640 4 set_mon_vals no callback set 2024-09-10T20:29:08.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 10 monclient: _renew_subs 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f7c1cef60 con 0x7f4f7c137e50 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 10 client.24236.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 10 monclient: _renew_subs 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f7c1255d0 con 0x7f4f7c137e50 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 1 librados: init done 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f7c139500 con 0x7f4f7c137e50 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 1 -- 172.21.15.153:0/3823750964 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f4f70013070 con 0x7f4f7c137e50 2024-09-10T20:29:08.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 1 --2- 172.21.15.153:0/3823750964 >> v2:172.21.15.176:6812/3003473398 conn(0x7f4f6803e590 0x7f4f68040a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 1 -- 172.21.15.153:0/3823750964 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(104..104 src has 1..104) ==== 6775+0+0 (secure 0 0 0) 0x7f4f70050500 con 0x7f4f7c137e50 2024-09-10T20:29:08.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 10 client.24236.objecter ms_dispatch 0x7f4f7c000ba0 osd_map(104..104 src has 1..104) 2024-09-10T20:29:08.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 3 client.24236.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T20:29:08.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 3 client.24236.objecter handle_osd_map decoding full epoch 104 2024-09-10T20:29:08.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f797fa640 20 client.24236.objecter dump_active .. 0 homeless 2024-09-10T20:29:08.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/3823750964 >> v2:172.21.15.176:6812/3003473398 conn(0x7f4f6803e590 0x7f4f68040a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.537+0000 7f4f7bfff640 10 monclient: get_auth_request con 0x7f4f6803e590 auth_method 0 2024-09-10T20:29:08.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.541+0000 7f4f797fa640 10 client.24236.objecter ms_handle_connect 0x7f4f6803e590 2024-09-10T20:29:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.541+0000 7f4f7bfff640 1 --2- 172.21.15.153:0/3823750964 >> v2:172.21.15.176:6812/3003473398 conn(0x7f4f6803e590 0x7f4f68040a50 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c000c10 tx=0x7f4f6c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.541+0000 7f4f797fa640 1 -- 172.21.15.153:0/3823750964 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f4f7004e730 con 0x7f4f7c137e50 2024-09-10T20:29:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.541+0000 7f4f797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:29:08.554 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.541+0000 7f4f797fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:29:08.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.769+0000 7f4f830a0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:29:08.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.769+0000 7f4f830a0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:29:08.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.769+0000 7f4f830a0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.769+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4f7c0a6be0 con 0x7f4f7c137e50 2024-09-10T20:29:08.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.769+0000 7f4f797fa640 1 -- 172.21.15.153:0/3823750964 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f4f70015480 con 0x7f4f7c137e50 2024-09-10T20:29:08.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.769+0000 7f4f797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:29:08.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.769+0000 7f4f797fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:29:08.780 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 10 librados: watch_flush enter 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 10 librados: watch_flush exit 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 20 client.24236.objecter shutdown clearing up homeless session... 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 10 client.24236.objecter successfully canceled tick 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 >> v2:172.21.15.176:6812/3003473398 conn(0x7f4f6803e590 msgr2=0x7f4f68040a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 --2- 172.21.15.153:0/3823750964 >> v2:172.21.15.176:6812/3003473398 conn(0x7f4f6803e590 0x7f4f68040a50 secure :-1 s=READY pgs=3261 cs=0 l=1 rev1=1 crypto rx=0x7f4f6c000c10 tx=0x7f4f6c008040 comp rx=0 tx=0).stop 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 10 monclient: shutdown 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c137e50 msgr2=0x7f4f7c138230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 --2- 172.21.15.153:0/3823750964 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c137e50 0x7f4f7c138230 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f4f70010c30 tx=0x7f4f7000e530 comp rx=0 tx=0).stop 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 shutdown_connections 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 --2- 172.21.15.153:0/3823750964 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f4f7c122330 0x7f4f7c122770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 --2- 172.21.15.153:0/3823750964 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f4f7c121990 0x7f4f7c121df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 --2- 172.21.15.153:0/3823750964 >> v2:172.21.15.176:6812/3003473398 conn(0x7f4f6803e590 0x7f4f68040a50 unknown :-1 s=CLOSED pgs=3261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 --2- 172.21.15.153:0/3823750964 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4f7c137e50 0x7f4f7c138230 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 >> 172.21.15.153:0/3823750964 conn(0x7f4f7c0aba00 msgr2=0x7f4f7c0a7070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:29:08.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 shutdown_connections 2024-09-10T20:29:08.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 -- 172.21.15.153:0/3823750964 wait complete. 2024-09-10T20:29:08.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.773+0000 7f4f830a0640 1 librados: shutdown 2024-09-10T20:29:08.798 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T20:29:08.798 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-10T20:29:08.798 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient: get_monmap_and_config 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient: build_initial_monmap 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient: monmap: 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:29:08.906160+0000 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:29:08.906160+0000 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.908 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:29:08.909 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:29:08.909 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:29:08.909 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:29:08.909 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:29:08.909 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 1 Processor -- start 2024-09-10T20:29:08.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 1 -- start start 2024-09-10T20:29:08.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient: init 2024-09-10T20:29:08.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient: _reopen_session rank -1 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient: _add_conns ranks=[2,3,0,1,4] 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3860 0x7f61b809d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient(hunting): picked mon.noname-b con 0x7f61b80a3860 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f61b80a3c40 0x7f61b80a4060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient(hunting): picked mon.noname-a con 0x7f61b80a3c40 addr v2:172.21.15.153:3300/0 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f61b8138640 addr v1:172.21.15.153:6789/0 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b80773d0 con 0x7f61b8138640 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f61b809cb40 con 0x7f61b80a3c40 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f61b809ce20 con 0x7f61b80a3860 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient(hunting): _renew_subs 2024-09-10T20:29:08.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61beb75640 10 monclient(hunting): authenticate will time out at 1109.045771s 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bd112640 1 -- >> v1:172.21.15.153:6789/0 conn(0x7f61b8138640 legacy=0x7f61b8138ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.153:6789/0 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61b7fff640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f61b80a3c40 msgr2=0x7f61b80a4060 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61b7fff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f61b80a3c40 0x7f61b80a4060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3860 0x7f61b809d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3860 0x7f61b809d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50362/0 (socket says 172.21.15.153:50362) 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 1 -- 172.21.15.153:0/1109771061 learned_addr learned my addr 172.21.15.153:0/1109771061 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient(hunting): get_auth_request con 0x7f61b80a3860 auth_method 0 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:08.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:29:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient(hunting): handle_auth_done global_id 24239 payload 274 2024-09-10T20:29:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 1 -- 172.21.15.153:0/1109771061 >> v2:172.21.15.153:3300/0 conn(0x7f61b80a3c40 msgr2=0x7f61b80a4060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 1 --2- 172.21.15.153:0/1109771061 >> v2:172.21.15.153:3300/0 conn(0x7f61b80a3c40 0x7f61b80a4060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 1 -- 172.21.15.153:0/1109771061 >> v1:172.21.15.153:6789/0 conn(0x7f61b8138640 legacy=0x7f61b8138ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient: _finish_hunting 0 2024-09-10T20:29:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 1 monclient: found mon.noname-b 2024-09-10T20:29:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:29:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.901+0000 7f61bc911640 1 -- 172.21.15.153:0/1109771061 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b809d100 con 0x7f61b80a3860 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61bc911640 1 -- 172.21.15.153:0/1109771061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3860 msgr2=0x7f61b809d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61bc911640 1 --2- 172.21.15.153:0/1109771061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3860 0x7f61b809d800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009870 tx=0x7f61ac01c4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12c58b7a837a55d1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 1 -- 172.21.15.153:0/1109771061 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0b0070 con 0x7f61b80a3860 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 10 monclient: dump: 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.918 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 10 monclient: _finish_auth 0 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 10 monclient: _check_auth_tickets 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 1 -- 172.21.15.153:0/1109771061 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61ac0ab600 con 0x7f61b80a3860 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 5 monclient: authenticate success, global_id 24239 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 10 monclient: get_monmap_and_config success 2024-09-10T20:29:08.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 4 set_mon_vals no callback set 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 1 -- 172.21.15.153:0/1109771061 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0ab9d0 con 0x7f61b80a3860 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61b77fe640 10 monclient: dump: 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.920 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 10 monclient: shutdown 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 1 -- 172.21.15.153:0/1109771061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3860 msgr2=0x7f61b809d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 1 --2- 172.21.15.153:0/1109771061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3860 0x7f61b809d800 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f61ac009870 tx=0x7f61ac01c4d0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 1 -- 172.21.15.153:0/1109771061 shutdown_connections 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 1 --2- 172.21.15.153:0/1109771061 >> v2:172.21.15.153:3300/0 conn(0x7f61b80a3c40 0x7f61b80a4060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 1 --2- 172.21.15.153:0/1109771061 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3860 0x7f61b809d800 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 1 -- 172.21.15.153:0/1109771061 >> 172.21.15.153:0/1109771061 conn(0x7f61b80a8510 msgr2=0x7f61b80a4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 1 -- 172.21.15.153:0/1109771061 shutdown_connections 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 1 -- 172.21.15.153:0/1109771061 wait complete. 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 10 monclient: build_initial_monmap 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.905+0000 7f61beb75640 10 monclient: monmap: 2024-09-10T20:29:08.921 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 librados: starting msgr at 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 librados: starting objecter 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 Processor -- start 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 -- start start 2024-09-10T20:29:08.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 librados: setting wanted keys 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 librados: calling monclient init 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient: init 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient: _reopen_session rank -1 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61b80a3860 0x7f61b8120a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient(hunting): picked mon.noname-b con 0x7f61b80a3860 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3c40 0x7f61b811e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient(hunting): picked mon.noname-c con 0x7f61b80a3c40 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b8138640 0x7f61b8137890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient(hunting): picked mon.noname-a con 0x7f61b8138640 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f61b8143240 con 0x7f61b8138640 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f61b81430c0 con 0x7f61b80a3860 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f61b81433c0 con 0x7f61b80a3c40 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient(hunting): _renew_subs 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient(hunting): authenticate will time out at 1109.052683s 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61bd112640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b8138640 msgr2=0x7f61b8137890 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61bd112640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b8138640 0x7f61b8137890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3c40 0x7f61b811e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3c40 0x7f61b811e960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50372/0 (socket says 172.21.15.153:50372) 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 -- 172.21.15.153:0/4134155204 learned_addr learned my addr 172.21.15.153:0/4134155204 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient(hunting): get_auth_request con 0x7f61b80a3c40 auth_method 0 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:08.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61bc911640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61b80a3860 0x7f61b8120a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient(hunting): handle_auth_done global_id 24242 payload 824 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61b80a3860 msgr2=0x7f61b8120a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61b80a3860 0x7f61b8120a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b8138640 msgr2=0x7f61b8137890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f61b8138640 0x7f61b8137890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient: _finish_hunting 0 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 monclient: found mon.noname-c 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 -- 172.21.15.153:0/4134155204 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b811fec0 con 0x7f61b80a3c40 2024-09-10T20:29:08.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61bc911640 10 monclient: get_auth_request con 0x7f61b80a3860 auth_method 0 2024-09-10T20:29:08.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61bc911640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f61b80a3860 0x7f61b8120a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3c40 msgr2=0x7f61b811e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b7fff640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3c40 0x7f61b811e960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f61a400b580 tx=0x7f61a400ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 client.?.objecter ms_handle_connect 0x7f61b80a3c40 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4003de0 con 0x7f61b80a3c40 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 monclient: dump: 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.930 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 monclient: _finish_auth 0 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 monclient: _check_auth_tickets 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a4015070 con 0x7f61b80a3c40 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 5 monclient: authenticate success, global_id 24242 2024-09-10T20:29:08.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient: _renew_subs 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b809f0b0 con 0x7f61b80a3c40 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a4005330 con 0x7f61b80a3c40 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b57fa640 10 monclient: dump: 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.932 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 client.24242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61b6ffd640 4 set_mon_vals no callback set 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient: _renew_subs 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b8121980 con 0x7f61b80a3c40 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.909+0000 7f61beb75640 1 librados: init done 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61beb75640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61beb75640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b8137dd0 0x7f61b81d0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61beb75640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f61b80a7900 con 0x7f61b8137dd0 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b8137dd0 0x7f61b81d0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 10 monclient: get_auth_request con 0x7f61b8137dd0 auth_method 0 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f61a40054d0 con 0x7f61b80a3c40 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 10 monclient(hunting): handle_auth_done global_id 24245 payload 824 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 1 -- 172.21.15.153:0/4134155204 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f61b809d880 con 0x7f61b8137dd0 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 1 --2- 172.21.15.153:0/4134155204 >> v2:172.21.15.176:6812/3003473398 conn(0x7f619c03ce50 0x7f619c03f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(104..104 src has 1..104) ==== 6775+0+0 (secure 0 0 0) 0x7f61a404fbe0 con 0x7f61b80a3c40 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 10 client.24242.objecter ms_dispatch 0x7f61b8000ba0 osd_map(104..104 src has 1..104) 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 3 client.24242.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 3 client.24242.objecter handle_osd_map decoding full epoch 104 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 20 client.24242.objecter dump_active .. 0 homeless 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bd112640 1 --2- 172.21.15.153:0/4134155204 >> v2:172.21.15.176:6812/3003473398 conn(0x7f619c03ce50 0x7f619c03f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bd112640 10 monclient: get_auth_request con 0x7f619c03ce50 auth_method 0 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b8137dd0 msgr2=0x7f61b81d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 10 client.24242.objecter ms_handle_connect 0x7f61b8137dd0 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 10 client.24242.objecter resend_mon_ops 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61bc911640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b8137dd0 0x7f61b81d0f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f61b80a5580 tx=0x7f61ac01ad10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0bb020 con 0x7f61b8137dd0 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 10 monclient: dump: 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f61ac0b6470 con 0x7f61b8137dd0 2024-09-10T20:29:08.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b8137dd0 msgr2=0x7f61b81d0f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b8137dd0 0x7f61b81d0f40 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f61b80a5580 tx=0x7f61ac01ad10 comp rx=0 tx=0).stop 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b8137dd0 msgr2=0x7f61b81d0f40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.913+0000 7f61b57fa640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b8137dd0 0x7f61b81d0f40 secure :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f61b80a5580 tx=0x7f61ac01ad10 comp rx=0 tx=0).stop 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.917+0000 7f61b57fa640 10 client.24242.objecter ms_handle_connect 0x7f619c03ce50 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.917+0000 7f61bd112640 1 --2- 172.21.15.153:0/4134155204 >> v2:172.21.15.176:6812/3003473398 conn(0x7f619c03ce50 0x7f619c03f310 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7f61b0003cc0 tx=0x7f61b00091f0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61beb75640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61beb75640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b802d450 0x7f61b802d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61beb75640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f61b813cf80 con 0x7f61b802d450 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b802d450 0x7f61b802d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:08.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 10 monclient: get_auth_request con 0x7f61b802d450 auth_method 0 2024-09-10T20:29:08.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:08.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:08.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:08.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:08.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:08.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:29:08.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 10 monclient(hunting): handle_auth_done global_id 24248 payload 824 2024-09-10T20:29:08.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 1 -- 172.21.15.153:0/4134155204 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f61b806a5b0 con 0x7f61b802d450 2024-09-10T20:29:08.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61b57fa640 10 client.24242.objecter ms_handle_connect 0x7f61b802d450 2024-09-10T20:29:08.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61b57fa640 10 client.24242.objecter resend_mon_ops 2024-09-10T20:29:08.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b802d450 msgr2=0x7f61b802d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:08.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61bc911640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b802d450 0x7f61b802d830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61ac01c300 tx=0x7f61ac0a8000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:08.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac0bb030 con 0x7f61b802d450 2024-09-10T20:29:08.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61b57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.921+0000 7f61b57fa640 10 monclient: dump: 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f61ac0b6dc0 con 0x7f61b802d450 2024-09-10T20:29:08.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61b57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b802d450 msgr2=0x7f61b802d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61b57fa640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b802d450 0x7f61b802d830 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f61ac01c300 tx=0x7f61ac0a8000 comp rx=0 tx=0).stop 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61b57fa640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b802d450 msgr2=0x7f61b802d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61b57fa640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b802d450 0x7f61b802d830 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:29:08.941 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 22, 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 50490, 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:29:08.942 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:29:08.943 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:29:08.944 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:29:08.945 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.946 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:29:08.947 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:29:08.948 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:29:08.949 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:08.950 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:08.951 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:29:08.952 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:29:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:29:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:08.953 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:29:08.953 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:29:08.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 10 librados: watch_flush enter 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 10 librados: watch_flush exit 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 20 client.24242.objecter shutdown clearing up homeless session... 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 10 client.24242.objecter successfully canceled tick 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 >> v2:172.21.15.176:6812/3003473398 conn(0x7f619c03ce50 msgr2=0x7f619c03f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 --2- 172.21.15.153:0/4134155204 >> v2:172.21.15.176:6812/3003473398 conn(0x7f619c03ce50 0x7f619c03f310 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7f61b0003cc0 tx=0x7f61b00091f0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 10 monclient: shutdown 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3c40 msgr2=0x7f61b811e960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3c40 0x7f61b811e960 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f61a400b580 tx=0x7f61a400ba40 comp rx=0 tx=0).stop 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61bd112640 1 -- 172.21.15.153:0/4134155204 reap_dead start 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 shutdown_connections 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 --2- 172.21.15.153:0/4134155204 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f61b80a3c40 0x7f61b811e960 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 >> 172.21.15.153:0/4134155204 conn(0x7f61b80a8510 msgr2=0x7f61b813fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:29:08.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 shutdown_connections 2024-09-10T20:29:08.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 -- 172.21.15.153:0/4134155204 wait complete. 2024-09-10T20:29:08.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:08.925+0000 7f61beb75640 1 librados: shutdown 2024-09-10T20:29:08.956 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T20:29:08.956 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T20:29:08.956 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:29:09.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient: get_monmap_and_config 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient: build_initial_monmap 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient: monmap: 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:29:09.066946+0000 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:29:09.066946+0000 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:29:09.069 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:29:09.070 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:29:09.070 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:09.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 1 Processor -- start 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 1 -- start start 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient: init 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient: _reopen_session rank -1 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient: _add_conns ranks=[3,2,4,0,1] 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0d980a41a0 0x7f0d980a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient(hunting): picked mon.noname-a con 0x7f0d980a41a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a4580 0x7f0d980a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient(hunting): picked mon.noname-b con 0x7f0d980a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f0d980a49a0 0x7f0d980a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient(hunting): picked mon.noname-c con 0x7f0d980a49a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f0d980a2570 con 0x7f0d980a41a0 2024-09-10T20:29:09.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f0d98070400 con 0x7f0d980a49a0 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0d980a6a20 con 0x7f0d980a4580 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient(hunting): _renew_subs 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d9d3e3640 10 monclient(hunting): authenticate will time out at 1109.206506s 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d96ffd640 1 -- >> v2:172.21.15.153:3300/0 conn(0x7f0d980a41a0 msgr2=0x7f0d980a3740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d96ffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0d980a41a0 0x7f0d980a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a4580 0x7f0d980a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a4580 0x7f0d980a59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50408/0 (socket says 172.21.15.153:50408) 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 1 -- 172.21.15.153:0/938267231 learned_addr learned my addr 172.21.15.153:0/938267231 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient(hunting): get_auth_request con 0x7f0d980a4580 auth_method 0 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:09.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d977fe640 1 --2- 172.21.15.153:0/938267231 >> v2:172.21.15.176:3300/0 conn(0x7f0d980a49a0 0x7f0d980a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d977fe640 10 monclient(hunting): get_auth_request con 0x7f0d980a49a0 auth_method 0 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d977fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d977fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient(hunting): handle_auth_done global_id 24251 payload 274 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 1 -- 172.21.15.153:0/938267231 >> v2:172.21.15.176:3300/0 conn(0x7f0d980a49a0 msgr2=0x7f0d980a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 1 --2- 172.21.15.153:0/938267231 >> v2:172.21.15.176:3300/0 conn(0x7f0d980a49a0 0x7f0d980a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 1 -- 172.21.15.153:0/938267231 >> v2:172.21.15.153:3300/0 conn(0x7f0d980a41a0 msgr2=0x7f0d980a3740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:29:09.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 1 --2- 172.21.15.153:0/938267231 >> v2:172.21.15.153:3300/0 conn(0x7f0d980a41a0 0x7f0d980a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient: _finish_hunting 0 2024-09-10T20:29:09.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 1 monclient: found mon.noname-b 2024-09-10T20:29:09.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:29:09.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:29:09.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.065+0000 7f0d967fc640 1 -- 172.21.15.153:0/938267231 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d980a3c80 con 0x7f0d980a4580 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d967fc640 1 -- 172.21.15.153:0/938267231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a4580 msgr2=0x7f0d980a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d967fc640 1 --2- 172.21.15.153:0/938267231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a4580 0x7f0d980a59d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0d980a7aa0 tx=0x7f0d8801c400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61e9f38e0543bdba server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 1 -- 172.21.15.153:0/938267231 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d880b0070 con 0x7f0d980a4580 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 10 monclient: dump: 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.079 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 10 monclient: _finish_auth 0 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d9d3e3640 5 monclient: authenticate success, global_id 24251 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d9d3e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 1 -- 172.21.15.153:0/938267231 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d880ab600 con 0x7f0d980a4580 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:29:09.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d9d3e3640 10 monclient: get_monmap_and_config success 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d9d3e3640 4 set_mon_vals no callback set 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 1 -- 172.21.15.153:0/938267231 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d880ab9d0 con 0x7f0d980a4580 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.069+0000 7f0d95ffb640 10 monclient: dump: 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:09.081 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient: shutdown 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/938267231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a4580 msgr2=0x7f0d980a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 --2- 172.21.15.153:0/938267231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a4580 0x7f0d980a59d0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0d980a7aa0 tx=0x7f0d8801c400 comp rx=0 tx=0).stop 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/938267231 shutdown_connections 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 --2- 172.21.15.153:0/938267231 >> v2:172.21.15.176:3300/0 conn(0x7f0d980a49a0 0x7f0d980a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 --2- 172.21.15.153:0/938267231 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a4580 0x7f0d980a59d0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 --2- 172.21.15.153:0/938267231 >> v2:172.21.15.153:3300/0 conn(0x7f0d980a41a0 0x7f0d980a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/938267231 >> 172.21.15.153:0/938267231 conn(0x7f0d980ab7e0 msgr2=0x7f0d980a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/938267231 shutdown_connections 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/938267231 wait complete. 2024-09-10T20:29:09.082 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient: build_initial_monmap 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient: monmap: 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 librados: starting msgr at 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 librados: starting objecter 2024-09-10T20:29:09.083 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 Processor -- start 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 -- start start 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 librados: setting wanted keys 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 librados: calling monclient init 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient: init 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient: _reopen_session rank -1 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a41a0 0x7f0d98128ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient(hunting): picked mon.noname-c con 0x7f0d980a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d980a4580 0x7f0d981269c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient(hunting): picked mon.noname-b con 0x7f0d980a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d980a49a0 0x7f0d9813e930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient(hunting): picked mon.noname-a con 0x7f0d980a49a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:29:09.084 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0d98142200 con 0x7f0d980a49a0 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0d980a6860 con 0x7f0d980a4580 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0d98142380 con 0x7f0d980a41a0 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient(hunting): _renew_subs 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d9d3e3640 10 monclient(hunting): authenticate will time out at 1109.213961s 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d977fe640 1 -- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d980a49a0 msgr2=0x7f0d9813e930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3300/0 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d977fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d980a49a0 0x7f0d9813e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a41a0 0x7f0d98128ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a41a0 0x7f0d98128ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:50414/0 (socket says 172.21.15.153:50414) 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 1 -- 172.21.15.153:0/3450512428 learned_addr learned my addr 172.21.15.153:0/3450512428 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient(hunting): get_auth_request con 0x7f0d980a41a0 auth_method 0 2024-09-10T20:29:09.085 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient(hunting): handle_auth_done global_id 24254 payload 824 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d980a4580 msgr2=0x7f0d981269c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d980a4580 0x7f0d981269c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d980a49a0 msgr2=0x7f0d9813e930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0d980a49a0 0x7f0d9813e930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 1 monclient: found mon.noname-c 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:29:09.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:29:09.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.073+0000 7f0d96ffd640 1 -- 172.21.15.153:0/3450512428 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d9812a2d0 con 0x7f0d980a41a0 2024-09-10T20:29:09.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d96ffd640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a41a0 msgr2=0x7f0d98128ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:09.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 client.?.objecter ms_handle_connect 0x7f0d980a41a0 2024-09-10T20:29:09.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:29:09.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d96ffd640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a41a0 0x7f0d98128ad0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00b550 tx=0x7f0d8c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0050f0 con 0x7f0d980a41a0 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 monclient: dump: 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:09.090 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 monclient: _finish_auth 0 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d8c015070 con 0x7f0d980a41a0 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d8c0054d0 con 0x7f0d980a41a0 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 monclient: dump: 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.091 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 5 monclient: authenticate success, global_id 24254 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d957fa640 4 set_mon_vals no callback set 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 10 monclient: _renew_subs 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d980a21c0 con 0x7f0d980a41a0 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 10 client.24254.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 10 monclient: _renew_subs 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:29:09.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d98129a30 con 0x7f0d980a41a0 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 1 librados: init done 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d98129c70 0x7f0d981d0d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0d980a3740 con 0x7f0d98129c70 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f0d8c005670 con 0x7f0d980a41a0 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 1 --2- 172.21.15.153:0/3450512428 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d8403ced0 0x7f0d8403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(104..104 src has 1..104) ==== 6775+0+0 (secure 0 0 0) 0x7f0d8c04fe30 con 0x7f0d980a41a0 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 10 client.24254.objecter ms_dispatch 0x7f0d98000ba0 osd_map(104..104 src has 1..104) 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 3 client.24254.objecter handle_osd_map got epochs [104,104] > 0 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 3 client.24254.objecter handle_osd_map decoding full epoch 104 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d98129c70 0x7f0d981d0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:09.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d7f7fe640 20 client.24254.objecter dump_active .. 0 homeless 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 10 monclient: get_auth_request con 0x7f0d98129c70 auth_method 0 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d977fe640 1 --2- 172.21.15.153:0/3450512428 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d8403ced0 0x7f0d8403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d977fe640 10 monclient: get_auth_request con 0x7f0d8403ced0 auth_method 0 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 10 monclient(hunting): handle_auth_done global_id 19717 payload 824 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.077+0000 7f0d967fc640 1 -- 172.21.15.153:0/3450512428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d980008e0 con 0x7f0d98129c70 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.081+0000 7f0d967fc640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d98129c70 msgr2=0x7f0d981d0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.081+0000 7f0d7f7fe640 10 client.24254.objecter ms_handle_connect 0x7f0d98129c70 2024-09-10T20:29:09.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.081+0000 7f0d7f7fe640 10 client.24254.objecter resend_mon_ops 2024-09-10T20:29:09.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.081+0000 7f0d967fc640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d98129c70 0x7f0d981d0d80 secure :-1 s=READY pgs=5502 cs=0 l=1 rev1=1 crypto rx=0x7f0d980a8a30 tx=0x7f0d880191a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:09.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.081+0000 7f0d977fe640 1 --2- 172.21.15.153:0/3450512428 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d8403ced0 0x7f0d8403f390 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f0d981283a0 tx=0x7f0d900098b0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:09.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.081+0000 7f0d7f7fe640 10 client.24254.objecter ms_handle_connect 0x7f0d8403ced0 2024-09-10T20:29:09.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.093+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d880ba030 con 0x7f0d98129c70 2024-09-10T20:29:09.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.093+0000 7f0d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.093+0000 7f0d7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.093+0000 7f0d7f7fe640 10 monclient: dump: 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:09.098 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:09.112 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.109+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d880b6420 con 0x7f0d98129c70 2024-09-10T20:29:09.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.109+0000 7f0d7f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:29:09.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.109+0000 7f0d7f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:29:09.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.109+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d98129c70 msgr2=0x7f0d981d0d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:09.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.109+0000 7f0d7f7fe640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d98129c70 0x7f0d981d0d80 secure :-1 s=READY pgs=5502 cs=0 l=1 rev1=1 crypto rx=0x7f0d980a8a30 tx=0x7f0d880191a0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.109+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d98129c70 msgr2=0x7f0d981d0d80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:29:09.113 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.109+0000 7f0d7f7fe640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d98129c70 0x7f0d981d0d80 secure :-1 s=CLOSED pgs=5502 cs=0 l=1 rev1=1 crypto rx=0x7f0d980a8a30 tx=0x7f0d880191a0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d9d3e3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:29:09.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d9d3e3640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 0x7f0d9802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:29:09.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d9d3e3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:29:09.123 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0d981d13e0 con 0x7f0d9813ee70 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d967fc640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 0x7f0d9802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d967fc640 10 monclient: get_auth_request con 0x7f0d9813ee70 auth_method 0 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d967fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d967fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d967fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d967fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.117+0000 7f0d967fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.121+0000 7f0d967fc640 10 monclient(hunting): handle_auth_done global_id 19720 payload 824 2024-09-10T20:29:09.124 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.121+0000 7f0d967fc640 1 -- 172.21.15.153:0/3450512428 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d98122d30 con 0x7f0d9813ee70 2024-09-10T20:29:09.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.125+0000 7f0d967fc640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 msgr2=0x7f0d9802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T20:29:09.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.125+0000 7f0d967fc640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 msgr2=0x7f0d9802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:29:09.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.125+0000 7f0d967fc640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 0x7f0d9802d450 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f0d88000c10 tx=0x7f0d880a8000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:29:09.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.125+0000 7f0d967fc640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 0x7f0d9802d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:29:09.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.329+0000 7f0d967fc640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 0x7f0d9802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:29:09.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.329+0000 7f0d967fc640 10 monclient: get_auth_request con 0x7f0d9813ee70 auth_method 0 2024-09-10T20:29:09.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.329+0000 7f0d967fc640 10 monclient: get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:29:09.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.329+0000 7f0d967fc640 10 monclient: _init_auth method 2 2024-09-10T20:29:09.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.329+0000 7f0d967fc640 10 monclient: _init_auth already have auth, reseting 2024-09-10T20:29:09.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.329+0000 7f0d967fc640 10 monclient: handle_auth_reply_more payload 9 2024-09-10T20:29:09.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.329+0000 7f0d967fc640 10 monclient: handle_auth_reply_more payload_len 9 2024-09-10T20:29:09.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.329+0000 7f0d967fc640 10 monclient: handle_auth_reply_more responding with 132 bytes 2024-09-10T20:29:09.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.329+0000 7f0d967fc640 10 monclient: handle_auth_done global_id 19720 payload 293 2024-09-10T20:29:09.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d967fc640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 msgr2=0x7f0d9802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:29:09.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d967fc640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 0x7f0d9802d450 secure :-1 s=READY pgs=5506 cs=0 l=1 rev1=1 crypto rx=0x7f0d8801a030 tx=0x7f0d880b4000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:29:09.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 10 client.24254.objecter ms_handle_connect 0x7f0d9813ee70 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 10 client.24254.objecter resend_mon_ops 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d880c6070 con 0x7f0d9813ee70 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 10 monclient: dump: 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:29:09.337 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:29:09.338 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:29:09.338 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:29:09.338 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:29:09.338 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:29:09.338 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:29:09.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f0d880c0dc0 con 0x7f0d9813ee70 2024-09-10T20:29:09.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:29:09.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:29:09.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 msgr2=0x7f0d9802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:09.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 0x7f0d9802d450 secure :-1 s=READY pgs=5506 cs=0 l=1 rev1=1 crypto rx=0x7f0d8801a030 tx=0x7f0d880b4000 comp rx=0 tx=0).stop 2024-09-10T20:29:09.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 msgr2=0x7f0d9802d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:29:09.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d7f7fe640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0d9813ee70 0x7f0d9802d450 secure :-1 s=CLOSED pgs=5506 cs=0 l=1 rev1=1 crypto rx=0x7f0d8801a030 tx=0x7f0d880b4000 comp rx=0 tx=0).stop 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 22, 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 195975, 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:29:09.340 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:29:09.341 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:29:09.342 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:29:09.343 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:29:09.344 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.345 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:29:09.346 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.347 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.348 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:09.349 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:29:09.350 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:29:09.351 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 10 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:29:09.352 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:29:09.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 10 librados: watch_flush enter 2024-09-10T20:29:09.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 10 librados: watch_flush exit 2024-09-10T20:29:09.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 20 client.24254.objecter shutdown clearing up homeless session... 2024-09-10T20:29:09.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 10 client.24254.objecter successfully canceled tick 2024-09-10T20:29:09.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d8403ced0 msgr2=0x7f0d8403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:09.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 --2- 172.21.15.153:0/3450512428 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0d8403ced0 0x7f0d8403f390 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7f0d981283a0 tx=0x7f0d900098b0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 10 monclient: shutdown 2024-09-10T20:29:09.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:29:09.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a41a0 msgr2=0x7f0d98128ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:29:09.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a41a0 0x7f0d98128ad0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d8c00b550 tx=0x7f0d8c00ba10 comp rx=0 tx=0).stop 2024-09-10T20:29:09.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d977fe640 1 -- 172.21.15.153:0/3450512428 reap_dead start 2024-09-10T20:29:09.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 shutdown_connections 2024-09-10T20:29:09.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 --2- 172.21.15.153:0/3450512428 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0d980a41a0 0x7f0d98128ad0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:29:09.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 >> 172.21.15.153:0/3450512428 conn(0x7f0d980ab7e0 msgr2=0x7f0d980a01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:29:09.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 shutdown_connections 2024-09-10T20:29:09.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 -- 172.21.15.153:0/3450512428 wait complete. 2024-09-10T20:29:09.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:29:09.333+0000 7f0d9d3e3640 1 librados: shutdown 2024-09-10T20:29:09.363 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T20:29:09.363 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-10T20:29:15.170 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: Running main() from gmock_main.cc 2024-09-10T20:29:15.170 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [----------] Global test environment set-up. 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2179 ms) 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3048 ms) 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3182 ms) 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (2995 ms) 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2302 ms) 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-10T20:29:15.171 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (2972 ms) 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3026 ms) 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3000 ms) 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (3102 ms) 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2932 ms) 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3974 ms) 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3508 ms) 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-10T20:29:15.172 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (2991 ms) 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (3481 ms) 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.Flush (3023 ms) 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3028 ms) 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2952 ms) 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3038 ms) 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3420 ms) 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3004 ms) 2024-09-10T20:29:15.173 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (22510 ms) 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (3646 ms) 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3029 ms) 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (22710 ms) 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3036 ms) 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3030 ms) 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [----------] 26 tests from LibRadosAio (119119 ms total) 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: 2024-09-10T20:29:15.174 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-10T20:29:19.433 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWritcPP (70183 ms total) 2024-09-10T20:29:19.433 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: 2024-09-10T20:29:19.433 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-10T20:29:19.433 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (193 ms) 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (193 ms total) 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (13 ms) 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (11 ms) 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (24 ms total) 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (52 ms) 2024-09-10T20:29:19.434 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (9 ms) 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (62 ms total) 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (65 ms) 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (9 ms) 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (74 ms total) 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1053 ms) 2024-09-10T20:29:19.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1053 ms total) 2024-09-10T20:29:19.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: 2024-09-10T20:29:19.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-10T20:29:19.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (123286 ms total) 2024-09-10T20:29:19.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28465 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28465 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27944 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 27944 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27976 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 27976 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28415 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28415 2024-09-10T20:29:19.437 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:29:19.438 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27936 2024-09-10T20:29:19.438 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 27936 2024-09-10T20:29:20.440 INFO:tasks.workunit.client.0.smithi176.stdout:pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (8052 ms) 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (14833 ms total) 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (23699 ms) 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4015 ms) 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (4237 ms) 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (31951 ms total) 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-10T20:29:20.441 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (124290 ms total) 2024-09-10T20:29:20.442 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-10T20:29:20.442 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-10T20:29:20.442 INFO:tasks.workunit.client.0.smithi176.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-10T20:29:31.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-10T20:29:31.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-10T20:29:31.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-10T20:29:31.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-10T20:29:31.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-10T20:29:31.435 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (3175 ms) 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (20524 ms) 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (6056 ms) 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (4464 ms) 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (2515 ms) 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3465 ms) 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2488 ms) 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-10T20:29:31.436 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3021 ms) 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (3025 ms) 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (2962 ms) 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (3031 ms) 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3424 ms) 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3002 ms) 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (7681 ms) 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-10T20:29:31.437 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3015 ms) 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2502 ms) 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (8305 ms) 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (5676 ms) 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3058 ms) 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (22662 ms) 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3027 ms) 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-10T20:29:31.438 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3227 ms) 2024-09-10T20:29:31.439 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-10T20:29:31.439 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3061 ms) 2024-09-10T20:29:31.439 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-10T20:29:31.439 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (3028 ms) 2024-09-10T20:29:31.439 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-10T20:29:31.439 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3015 ms) 2024-09-10T20:29:31.439 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-10T20:29:31.439 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2945 ms) 2024-09-10T20:29:31.439 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-10T20:29:36.701 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAi api_list: : cursor()=7:5c6b0b28:::7:head expected=7:5c6b0b28:::7:head 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:5c6b0b28:::7:head -> 7 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=7 expected=7 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:02547ec2:::1:head 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:02547ec2:::1:head expected=7:02547ec2:::1:head 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:02547ec2:::1:head -> 1 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=1 expected=1 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:c4fdafeb:::6:head 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:c4fdafeb:::6:head expected=7:c4fdafeb:::6:head 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:c4fdafeb:::6:head -> 6 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=6 expected=6 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:bd63b0f1:::8:head 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:bd63b0f1:::8:head expected=7:bd63b0f1:::8:head 2024-09-10T20:29:36.702 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:bd63b0f1:::8:head -> 8 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=8 expected=8 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:b29083e3:::5:head 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:b29083e3:::5:head expected=7:b29083e3:::5:head 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:b29083e3:::5:head -> 5 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=5 expected=5 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:de5d7c5f:::12:head 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:de5d7c5f:::12:head expected=7:de5d7c5f:::12:head 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:de5d7c5f:::12:head -> 12 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=12 expected=12 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:d83876eb:::4:head 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:d83876eb:::4:head expected=7:d83876eb:::4:head 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:d83876eb:::4:head -> 4 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=4 expected=4 2024-09-10T20:29:36.703 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : seek to 7:cfc208b3:::3:head 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : cursor()=7:cfc208b3:::3:head expected=7:cfc208b3:::3:head 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: > 7:cfc208b3:::3:head -> 3 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: : entry=3 expected=3 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (668 ms) 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (126038 ms) 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: split 0/5 -> MIN 7:33333333::::head 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: split 1/5 -> 7:33333333::::head 7:66666666::::head 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: split 2/5 -> 7:66666666::::head 7:99999999::::head 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: split 3/5 -> 7:99999999::::head 7:cccccccc::::head 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: split 4/5 -> 7:cccccccc::::head MAX 2024-09-10T20:29:36.704 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (5963 ms) 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [----------] 7 tests from LibRadosList (133295 ms total) 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1083 ms) 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: myset foo1,foo2,foo3 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo1 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo2 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo3 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: myset foo1,foo4,foo5 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo4 2024-09-10T20:29:36.705 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo5 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo1 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: myset foo6,foo7 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo7 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: foo6 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns1:foo4 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns1:foo5 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns2:foo7 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns2:foo6 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: ns1:foo1 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: :foo1 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: :foo2 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: :foo3 2024-09-10T20:29:36.706 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (96 ms) 2024-09-10T20:29:36.707 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-10T20:29:36.707 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 1 0 2024-09-10T20:29:36.707 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 10 0 2024-09-10T20:29:36.707 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 13 0 2024-09-10T20:29:42.475 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 7 0 2024-09-10T20:29:42.475 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 14 0 2024-09-10T20:29:42.475 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 0 0 2024-09-10T20:29:42.475 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 15 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 11 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 5 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 8 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 6 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 3 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 4 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 12 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 9 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 2 0 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (158 ms) 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [----------] 3 tests from LibRadosListEC (1338 ms total) 2024-09-10T20:29:42.476 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 2024-09-10T20:29:42.477 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-10T20:29:42.477 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-10T20:29:42.477 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3023 ms) 2024-09-10T20:29:42.477 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [----------] 1 test from LibRadosListNP (3023 ms total) 2024-09-10T20:29:42.477 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: 2024-09-10T20:29:42.477 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [----------] Global test environment tear-down 2024-09-10T20:29:42.477 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [==========] 11 tests from 3 test suites ran. (146369 ms total) 2024-09-10T20:29:42.477 INFO:tasks.workunit.client.0.smithi176.stdout: api_list: [ PASSED ] 11 tests. 2024-09-10T20:30:39.364 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-10T20:30:39.365 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-10T20:30:39.365 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-10T20:30:39.369 INFO:tasks.ceph.mon.a:Started 2024-09-10T20:30:39.369 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T20:30:39.369 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:30:39.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient: get_monmap_and_config 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient: build_initial_monmap 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient: monmap: 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:39.487832+0000 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:39.487832+0000 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:39.490 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:39.491 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:39.491 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 1 Processor -- start 2024-09-10T20:30:39.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 1 -- start start 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient: init 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient: _add_conns ranks=[4,3,1,2,0] 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7ff9440a5c50 0x7ff9440a6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient(hunting): picked mon.noname-c con 0x7ff9440a5c50 addr v2:172.21.15.176:3300/0 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 0x7ff9440a6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient(hunting): picked mon.noname-a con 0x7ff9440a54d0 addr v2:172.21.15.153:3300/0 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff944138a10 addr v1:172.21.15.176:6789/0 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9440a2960 con 0x7ff944138a10 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7ff9440a6ab0 con 0x7ff9440a54d0 2024-09-10T20:30:39.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7ff9440a2f20 con 0x7ff9440a5c50 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9492f3640 10 monclient(hunting): authenticate will time out at 1199.627160s 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 0x7ff9440a6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 0x7ff9440a6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:52892/0 (socket says 172.21.15.153:52892) 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 1 -- 172.21.15.153:0/1658193613 learned_addr learned my addr 172.21.15.153:0/1658193613 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 10 monclient(hunting): get_auth_request con 0x7ff9440a54d0 auth_method 0 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 1 -- 172.21.15.153:0/1658193613 >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 msgr2=0x7ff9440a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 1 -- 172.21.15.153:0/1658193613 >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 msgr2=0x7ff9440a6570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:30:39.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 1 --2- 172.21.15.153:0/1658193613 >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 0x7ff9440a6570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff9427fc640 1 --2- 172.21.15.153:0/1658193613 >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 0x7ff9440a6570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff942ffd640 1 --2- 172.21.15.153:0/1658193613 >> v2:172.21.15.176:3300/0 conn(0x7ff9440a5c50 0x7ff9440a6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff942ffd640 10 monclient(hunting): get_auth_request con 0x7ff9440a5c50 auth_method 0 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff942ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff942ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff942ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff942ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff942ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.484+0000 7ff942ffd640 10 monclient(hunting): handle_auth_done global_id 19963 payload 274 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.488+0000 7ff942ffd640 1 -- 172.21.15.153:0/1658193613 >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 msgr2=0x7ff9440a6570 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.488+0000 7ff942ffd640 1 --2- 172.21.15.153:0/1658193613 >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 0x7ff9440a6570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.496 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.488+0000 7ff942ffd640 1 -- 172.21.15.153:0/1658193613 >> v1:172.21.15.176:6789/0 conn(0x7ff944138a10 legacy=0x7ff944138e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.488+0000 7ff942ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:30:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.488+0000 7ff942ffd640 1 monclient: found mon.noname-c 2024-09-10T20:30:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.488+0000 7ff942ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.488+0000 7ff942ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:30:39.497 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.488+0000 7ff942ffd640 1 -- 172.21.15.153:0/1658193613 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9440a2c40 con 0x7ff9440a5c50 2024-09-10T20:30:39.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 -- 172.21.15.153:0/1658193613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9440a5c50 msgr2=0x7ff9440a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:39.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 --2- 172.21.15.153:0/1658193613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9440a5c50 0x7ff9440a6030 secure :-1 s=READY pgs=5625 cs=0 l=1 rev1=1 crypto rx=0x7ff94406a8e0 tx=0x7ff93401c3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53c4e7ddf992fcc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:39.501 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 1 -- 172.21.15.153:0/1658193613 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9340b0070 con 0x7ff9440a5c50 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 10 monclient: dump: 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:39.502 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 10 monclient: _finish_auth 0 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 1 -- 172.21.15.153:0/1658193613 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9340ab600 con 0x7ff9440a5c50 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 1 -- 172.21.15.153:0/1658193613 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9340ab9d0 con 0x7ff9440a5c50 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff941ffb640 10 monclient: dump: 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:39.503 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 5 monclient: authenticate success, global_id 19963 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 4 set_mon_vals no callback set 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient: shutdown 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 -- 172.21.15.153:0/1658193613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9440a5c50 msgr2=0x7ff9440a6030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 --2- 172.21.15.153:0/1658193613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9440a5c50 0x7ff9440a6030 secure :-1 s=READY pgs=5625 cs=0 l=1 rev1=1 crypto rx=0x7ff94406a8e0 tx=0x7ff93401c3d0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 -- 172.21.15.153:0/1658193613 shutdown_connections 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 --2- 172.21.15.153:0/1658193613 >> v2:172.21.15.153:3300/0 conn(0x7ff9440a54d0 0x7ff9440a6570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 --2- 172.21.15.153:0/1658193613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff9440a5c50 0x7ff9440a6030 unknown :-1 s=CLOSED pgs=5625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 -- 172.21.15.153:0/1658193613 >> 172.21.15.153:0/1658193613 conn(0x7ff9440a7d50 msgr2=0x7ff9440a8160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 -- 172.21.15.153:0/1658193613 shutdown_connections 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 -- 172.21.15.153:0/1658193613 wait complete. 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient: build_initial_monmap 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient: monmap: 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:39.505 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 librados: starting msgr at 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 librados: starting objecter 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 Processor -- start 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 -- start start 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 librados: setting wanted keys 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 librados: calling monclient init 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient: init 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9440a54d0 0x7ff944137420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:39.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient(hunting): picked mon.noname-c con 0x7ff9440a54d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9440a5c50 0x7ff944137960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient(hunting): picked mon.noname-a con 0x7ff9440a5c50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff944138a10 0x7ff94413d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient(hunting): picked mon.noname-b con 0x7ff944138a10 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff944143820 con 0x7ff9440a5c50 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441436a0 con 0x7ff944138a10 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff9441439a0 con 0x7ff9440a54d0 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9492f3640 10 monclient(hunting): authenticate will time out at 1199.634914s 2024-09-10T20:30:39.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9440a54d0 0x7ff944137420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9440a54d0 0x7ff944137420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:47898/0 (socket says 172.21.15.153:47898) 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9427fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9440a5c50 0x7ff944137960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 -- 172.21.15.153:0/3605580611 learned_addr learned my addr 172.21.15.153:0/3605580611 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient(hunting): get_auth_request con 0x7ff9440a54d0 auth_method 0 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9427fc640 10 monclient(hunting): get_auth_request con 0x7ff9440a5c50 auth_method 0 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9427fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9427fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:39.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9427fc640 1 -- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9440a5c50 msgr2=0x7ff944137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9427fc640 1 -- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9440a5c50 msgr2=0x7ff944137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9427fc640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9440a5c50 0x7ff944137960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9427fc640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9440a5c50 0x7ff944137960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9437fe640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff944138a10 0x7ff94413d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9437fe640 10 monclient(hunting): get_auth_request con 0x7ff944138a10 auth_method 0 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff9437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient(hunting): handle_auth_done global_id 24431 payload 824 2024-09-10T20:30:39.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 -- 172.21.15.153:0/3605580611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff944138a10 msgr2=0x7ff94413d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff944138a10 0x7ff94413d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 -- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9440a5c50 msgr2=0x7ff944137960 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9440a5c50 0x7ff944137960 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 monclient: found mon.noname-c 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.492+0000 7ff942ffd640 1 -- 172.21.15.153:0/3605580611 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff944122000 con 0x7ff9440a54d0 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff942ffd640 1 -- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9440a54d0 msgr2=0x7ff944137420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff942ffd640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9440a54d0 0x7ff944137420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff9340ae040 tx=0x7ff934019170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 client.?.objecter ms_handle_connect 0x7ff9440a54d0 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:39.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 1 -- 172.21.15.153:0/3605580611 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9340b8020 con 0x7ff9440a54d0 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 monclient: dump: 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:39.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 5 monclient: authenticate success, global_id 24431 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 1 -- 172.21.15.153:0/3605580611 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9340b4840 con 0x7ff9440a54d0 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 1 -- 172.21.15.153:0/3605580611 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9340b0040 con 0x7ff9440a54d0 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 monclient: dump: 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:39.512 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:39.513 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:39.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9417fa640 4 set_mon_vals no callback set 2024-09-10T20:30:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 10 monclient: _renew_subs 2024-09-10T20:30:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9441222e0 con 0x7ff9440a54d0 2024-09-10T20:30:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 10 client.24431.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 10 monclient: _renew_subs 2024-09-10T20:30:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff944140180 con 0x7ff9440a54d0 2024-09-10T20:30:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 1 librados: init done 2024-09-10T20:30:39.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff944138e20 con 0x7ff9440a54d0 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 1 -- 172.21.15.153:0/3605580611 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7ff9340bb020 con 0x7ff9440a54d0 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 1 --2- 172.21.15.153:0/3605580611 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff93003ced0 0x7ff93003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 1 -- 172.21.15.153:0/3605580611 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(195..195 src has 1..195) ==== 5216+0+0 (secure 0 0 0) 0x7ff9340e9620 con 0x7ff9440a54d0 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 10 client.24431.objecter ms_dispatch 0x7ff944000ba0 osd_map(195..195 src has 1..195) 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 3 client.24431.objecter handle_osd_map got epochs [195,195] > 0 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 3 client.24431.objecter handle_osd_map decoding full epoch 195 2024-09-10T20:30:39.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.496+0000 7ff92b7fe640 20 client.24431.objecter dump_active .. 0 homeless 2024-09-10T20:30:39.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.500+0000 7ff9427fc640 1 --2- 172.21.15.153:0/3605580611 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff93003ced0 0x7ff93003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:39.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.500+0000 7ff9427fc640 10 monclient: get_auth_request con 0x7ff93003ced0 auth_method 0 2024-09-10T20:30:39.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.500+0000 7ff9427fc640 1 --2- 172.21.15.153:0/3605580611 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff93003ced0 0x7ff93003f390 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7ff94413f9c0 tx=0x7ff938008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:39.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.500+0000 7ff92b7fe640 10 client.24431.objecter ms_handle_connect 0x7ff93003ced0 2024-09-10T20:30:39.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.504+0000 7ff92b7fe640 1 -- 172.21.15.153:0/3605580611 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7ff9340c2030 con 0x7ff9440a54d0 2024-09-10T20:30:39.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.504+0000 7ff92b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:39.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.504+0000 7ff92b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:30:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:30:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff9440a6df0 con 0x7ff9440a54d0 2024-09-10T20:30:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff92b7fe640 1 -- 172.21.15.153:0/3605580611 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7ff934016050 con 0x7ff9440a54d0 2024-09-10T20:30:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff92b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:30:39.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff92b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:39.742 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:30:39.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 10 librados: watch_flush enter 2024-09-10T20:30:39.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 10 librados: watch_flush exit 2024-09-10T20:30:39.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 20 client.24431.objecter shutdown clearing up homeless session... 2024-09-10T20:30:39.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 10 client.24431.objecter successfully canceled tick 2024-09-10T20:30:39.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff93003ced0 msgr2=0x7ff93003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:39.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 1 --2- 172.21.15.153:0/3605580611 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff93003ced0 0x7ff93003f390 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7ff94413f9c0 tx=0x7ff938008040 comp rx=0 tx=0).stop 2024-09-10T20:30:39.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 10 monclient: shutdown 2024-09-10T20:30:39.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:39.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9440a54d0 msgr2=0x7ff944137420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.736+0000 7ff9492f3640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9440a54d0 0x7ff944137420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff9340ae040 tx=0x7ff934019170 comp rx=0 tx=0).stop 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.740+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 shutdown_connections 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.740+0000 7ff9492f3640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff944138a10 0x7ff94413d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.740+0000 7ff9492f3640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff9440a5c50 0x7ff944137960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.740+0000 7ff9492f3640 1 --2- 172.21.15.153:0/3605580611 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff93003ced0 0x7ff93003f390 unknown :-1 s=CLOSED pgs=3311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.740+0000 7ff9492f3640 1 --2- 172.21.15.153:0/3605580611 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff9440a54d0 0x7ff944137420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.740+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 >> 172.21.15.153:0/3605580611 conn(0x7ff9440a7d50 msgr2=0x7ff94406ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.740+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 shutdown_connections 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.740+0000 7ff9492f3640 1 -- 172.21.15.153:0/3605580611 wait complete. 2024-09-10T20:30:39.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:39.740+0000 7ff9492f3640 1 librados: shutdown 2024-09-10T20:30:42.759 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:30:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient: get_monmap_and_config 2024-09-10T20:30:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient: build_initial_monmap 2024-09-10T20:30:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient: monmap: 2024-09-10T20:30:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:42.889 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:42.887174+0000 2024-09-10T20:30:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:42.887174+0000 2024-09-10T20:30:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:42.890 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:42.890 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:42.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 1 Processor -- start 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 1 -- start start 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient: init 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient: _add_conns ranks=[4,2,3,1,0] 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f10ec0a28c0 0x7f10ec09d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient(hunting): picked mon.noname-c con 0x7f10ec0a28c0 addr v2:172.21.15.176:3300/0 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec0a3060 0x7f10ec09d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient(hunting): picked mon.noname-b con 0x7f10ec0a3060 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 0x7f10ec135580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient(hunting): picked mon.noname-a con 0x7f10ec09dab0 addr v2:172.21.15.153:3300/0 2024-09-10T20:30:42.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f10ec09f640 con 0x7f10ec09dab0 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f10ec0a34e0 con 0x7f10ec0a28c0 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f10ec0a2ca0 con 0x7f10ec0a3060 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10f1bd0640 10 monclient(hunting): authenticate will time out at 1203.026860s 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 0x7f10ec135580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec0a3060 0x7f10ec09d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 0x7f10ec135580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49628/0 (socket says 172.21.15.153:49628) 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient(hunting): get_auth_request con 0x7f10ec0a3060 auth_method 0 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 1 -- 172.21.15.153:0/1991965078 learned_addr learned my addr 172.21.15.153:0/1991965078 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:42.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 10 monclient(hunting): get_auth_request con 0x7f10ec09dab0 auth_method 0 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 1 -- 172.21.15.153:0/1991965078 >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 msgr2=0x7f10ec135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 1 -- 172.21.15.153:0/1991965078 >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 msgr2=0x7f10ec135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:30:42.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 1 --2- 172.21.15.153:0/1991965078 >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 0x7f10ec135580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10ebfff640 1 --2- 172.21.15.153:0/1991965078 >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 0x7f10ec135580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/1991965078 >> v2:172.21.15.176:3300/0 conn(0x7f10ec0a28c0 0x7f10ec09d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eb7fe640 10 monclient(hunting): get_auth_request con 0x7f10ec0a28c0 auth_method 0 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient(hunting): handle_auth_done global_id 24440 payload 274 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 1 -- 172.21.15.153:0/1991965078 >> v2:172.21.15.176:3300/0 conn(0x7f10ec0a28c0 msgr2=0x7f10ec09d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 1 --2- 172.21.15.153:0/1991965078 >> v2:172.21.15.176:3300/0 conn(0x7f10ec0a28c0 0x7f10ec09d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 1 -- 172.21.15.153:0/1991965078 >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 msgr2=0x7f10ec135580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 1 --2- 172.21.15.153:0/1991965078 >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 0x7f10ec135580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient: _finish_hunting 0 2024-09-10T20:30:42.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 1 monclient: found mon.noname-b 2024-09-10T20:30:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:30:42.898 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.884+0000 7f10eaffd640 1 -- 172.21.15.153:0/1991965078 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ec09f890 con 0x7f10ec0a3060 2024-09-10T20:30:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.888+0000 7f10eaffd640 1 -- 172.21.15.153:0/1991965078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec0a3060 msgr2=0x7f10ec09d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:42.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.888+0000 7f10eaffd640 1 --2- 172.21.15.153:0/1991965078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec0a3060 0x7f10ec09d570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f10ec06bfc0 tx=0x7f10e401e750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf729b297c89e810 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.888+0000 7f10ea7fc640 1 -- 172.21.15.153:0/1991965078 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e40b4070 con 0x7f10ec0a3060 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.888+0000 7f10ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.888+0000 7f10ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.888+0000 7f10ea7fc640 10 monclient: dump: 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:42.906 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ea7fc640 10 monclient: _finish_auth 0 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ea7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ea7fc640 1 -- 172.21.15.153:0/1991965078 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e40a77e0 con 0x7f10ec0a3060 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 5 monclient: authenticate success, global_id 24440 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ea7fc640 1 -- 172.21.15.153:0/1991965078 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e40a7bb0 con 0x7f10ec0a3060 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 4 set_mon_vals no callback set 2024-09-10T20:30:42.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ea7fc640 10 monclient: dump: 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient: shutdown 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:42.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 -- 172.21.15.153:0/1991965078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec0a3060 msgr2=0x7f10ec09d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/1991965078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec0a3060 0x7f10ec09d570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f10ec06bfc0 tx=0x7f10e401e750 comp rx=0 tx=0).stop 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 -- 172.21.15.153:0/1991965078 shutdown_connections 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/1991965078 >> v2:172.21.15.153:3300/0 conn(0x7f10ec09dab0 0x7f10ec135580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/1991965078 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec0a3060 0x7f10ec09d570 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/1991965078 >> v2:172.21.15.176:3300/0 conn(0x7f10ec0a28c0 0x7f10ec09d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 -- 172.21.15.153:0/1991965078 >> 172.21.15.153:0/1991965078 conn(0x7f10ec0a71b0 msgr2=0x7f10ec0a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 -- 172.21.15.153:0/1991965078 shutdown_connections 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 -- 172.21.15.153:0/1991965078 wait complete. 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient: build_initial_monmap 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient: monmap: 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:42.909 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 librados: starting msgr at 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 librados: starting objecter 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 Processor -- start 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 -- start start 2024-09-10T20:30:42.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 librados: setting wanted keys 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 librados: calling monclient init 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient: init 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 0x7f10ec13abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient(hunting): picked mon.noname-c con 0x7f10ec09dab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10ec12d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient(hunting): picked mon.noname-a con 0x7f10ec0a28c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 0x7f10ec12dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient(hunting): picked mon.noname-b con 0x7f10ec0a3060 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec0a1b80 con 0x7f10ec0a28c0 2024-09-10T20:30:42.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f10ec0a1a00 con 0x7f10ec0a3060 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f10ec0a1d00 con 0x7f10ec09dab0 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10f1bd0640 10 monclient(hunting): authenticate will time out at 1203.034201s 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 0x7f10ec13abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10ec12d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10ec12d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49630/0 (socket says 172.21.15.153:49630) 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 0x7f10ec13abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44722/0 (socket says 172.21.15.153:44722) 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 1 -- 172.21.15.153:0/2784488568 learned_addr learned my addr 172.21.15.153:0/2784488568 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 10 monclient(hunting): get_auth_request con 0x7f10ec0a28c0 auth_method 0 2024-09-10T20:30:42.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient(hunting): get_auth_request con 0x7f10ec09dab0 auth_method 0 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 msgr2=0x7f10ec12d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 msgr2=0x7f10ec12d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10ec12d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:42.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eaffd640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10ec12d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ebfff640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 0x7f10ec12dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient(hunting): handle_auth_done global_id 24443 payload 824 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 msgr2=0x7f10ec12dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 0x7f10ec12dad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 msgr2=0x7f10ec12d590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10ec12d590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 1 monclient: found mon.noname-c 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10eb7fe640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10ec051410 con 0x7f10ec09dab0 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ebfff640 10 monclient: get_auth_request con 0x7f10ec0a3060 auth_method 0 2024-09-10T20:30:42.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.892+0000 7f10ebfff640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 0x7f10ec12dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 msgr2=0x7f10ec13abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 0x7f10ec13abb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f10d4007e60 tx=0x7f10d400e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 client.?.objecter ms_handle_connect 0x7f10ec09dab0 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d4013790 con 0x7f10ec09dab0 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: dump: 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.915 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: _finish_auth 0 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: _check_auth_tickets 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 5 monclient: authenticate success, global_id 24443 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 10 monclient: _renew_subs 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10d4019070 con 0x7f10ec09dab0 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:42.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10ec13b470 con 0x7f10ec09dab0 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10d4013b50 con 0x7f10ec09dab0 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: dump: 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:42.917 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e9ffb640 4 set_mon_vals no callback set 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 10 client.24443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 10 monclient: _renew_subs 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f10ec0741c0 con 0x7f10ec09dab0 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 1 librados: init done 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 msgr2=0x7f10ec13abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 msgr2=0x7f10ec13abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 0x7f10ec13abb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f10d4007e60 tx=0x7f10d400e4b0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 0x7f10ec13abb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f10d4007e60 tx=0x7f10d400e4b0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:42.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:42.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 msgr2=0x7f10ec13abb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:42.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10ec09dab0 0x7f10ec13abb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:30:42.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10d0007290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient(hunting): picked mon.a con 0x7f10ec0a28c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:42.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d0007c60 0x7f10d000a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient(hunting): picked mon.c con 0x7f10d0007c60 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 0x7f10d000cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient(hunting): picked mon.b con 0x7f10ec0a3060 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f10d000d550 con 0x7f10ec0a28c0 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f10d000d6d0 con 0x7f10ec0a3060 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f10d000d850 con 0x7f10d0007c60 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10e3fff640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d0007c60 0x7f10d000a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10d0007290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient(hunting): get_auth_request con 0x7f10d0007c60 auth_method 0 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 10 monclient(hunting): get_auth_request con 0x7f10ec0a28c0 auth_method 0 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:42.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10f1bd0640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient(hunting): handle_auth_done global_id 24443 payload 293 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 msgr2=0x7f10d000cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10ebfff640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 0x7f10d000cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 0x7f10d000cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 msgr2=0x7f10d0007290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10d0007290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient: _finish_hunting 0 2024-09-10T20:30:42.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 1 monclient: found mon.c 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f10d000db70 con 0x7f10d0007c60 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ec135520 con 0x7f10d0007c60 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10ec09f890 con 0x7f10d0007c60 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient: _finish_auth 0 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 10 monclient: _check_auth_tickets 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10ebfff640 10 monclient: get_auth_request con 0x7f10ec0a3060 auth_method 0 2024-09-10T20:30:42.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10d0007290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.896+0000 7f10ebfff640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 0x7f10d000cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 10 client.24443.objecter ms_handle_connect 0x7f10d0007c60 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 10 client.24443.objecter resend_mon_ops 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10eaffd640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d0007c60 msgr2=0x7f10d000a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10eaffd640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d0007c60 0x7f10d000a0f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f10e40a7950 tx=0x7f10d40098d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10e40a75e0 con 0x7f10d0007c60 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 10 monclient: dump: 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.924 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10e40be020 con 0x7f10d0007c60 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e97fa640 4 set_mon_vals no callback set 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f10e40a7a00 con 0x7f10d0007c60 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 1 --2- 172.21.15.153:0/2784488568 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10ec09dab0 0x7f10d0047760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 4 ==== osd_map(199..199 src has 1..199) ==== 5538+0+0 (secure 0 0 0) 0x7f10e40e7fb0 con 0x7f10d0007c60 2024-09-10T20:30:42.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 10 client.24443.objecter ms_dispatch 0x7f10ec000ba0 osd_map(199..199 src has 1..199) 2024-09-10T20:30:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 3 client.24443.objecter handle_osd_map got epochs [199,199] > 0 2024-09-10T20:30:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 3 client.24443.objecter handle_osd_map decoding full epoch 199 2024-09-10T20:30:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10e3fff640 20 client.24443.objecter dump_active .. 0 homeless 2024-09-10T20:30:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/2784488568 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10ec09dab0 0x7f10d0047760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:42.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.900+0000 7f10eb7fe640 10 monclient: get_auth_request con 0x7f10ec09dab0 auth_method 0 2024-09-10T20:30:42.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.904+0000 7f10eb7fe640 1 --2- 172.21.15.153:0/2784488568 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10ec09dab0 0x7f10d0047760 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f10d4007fd0 tx=0x7f10d400b030 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:42.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.904+0000 7f10e3fff640 10 client.24443.objecter ms_handle_connect 0x7f10ec09dab0 2024-09-10T20:30:42.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.904+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f10e4116cf0 con 0x7f10d0007c60 2024-09-10T20:30:42.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.904+0000 7f10e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:42.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.904+0000 7f10e3fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:42.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.908+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f10e4018720 con 0x7f10d0007c60 2024-09-10T20:30:42.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:42.908+0000 7f10e3fff640 10 monclient: handle_mon_command_ack 1 not found 2024-09-10T20:30:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.136+0000 7f10f1bd0640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:30:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.136+0000 7f10f1bd0640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:30:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.136+0000 7f10f1bd0640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.136+0000 7f10f1bd0640 1 -- 172.21.15.153:0/2784488568 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f10ec13b760 con 0x7f10d0007c60 2024-09-10T20:30:43.143 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10e3fff640 1 -- 172.21.15.153:0/2784488568 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7f10e40ae3a0 con 0x7f10d0007c60 2024-09-10T20:30:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:30:43.144 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10e3fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:43.144 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":22,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":94,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 10 librados: watch_flush enter 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 10 librados: watch_flush exit 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 20 client.24443.objecter shutdown clearing up homeless session... 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 10 client.24443.objecter successfully canceled tick 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 -- 172.21.15.153:0/2784488568 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10ec09dab0 msgr2=0x7f10d0047760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/2784488568 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10ec09dab0 0x7f10d0047760 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f10d4007fd0 tx=0x7f10d400b030 comp rx=0 tx=0).stop 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 10 monclient: shutdown 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 -- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d0007c60 msgr2=0x7f10d000a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d0007c60 0x7f10d000a0f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f10e40a7950 tx=0x7f10d40098d0 comp rx=0 tx=0).stop 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 -- 172.21.15.153:0/2784488568 shutdown_connections 2024-09-10T20:30:43.147 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f10ec0a3060 0x7f10d000cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:43.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/2784488568 >> v2:172.21.15.176:6812/3003473398 conn(0x7f10ec09dab0 0x7f10d0047760 unknown :-1 s=CLOSED pgs=3313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:43.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f10d0007c60 0x7f10d000a0f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:43.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 --2- 172.21.15.153:0/2784488568 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f10ec0a28c0 0x7f10d0007290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:43.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 -- 172.21.15.153:0/2784488568 >> 172.21.15.153:0/2784488568 conn(0x7f10ec0a71b0 msgr2=0x7f10ec131b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:43.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 -- 172.21.15.153:0/2784488568 shutdown_connections 2024-09-10T20:30:43.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 -- 172.21.15.153:0/2784488568 wait complete. 2024-09-10T20:30:43.148 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:43.140+0000 7f10f1bd0640 1 librados: shutdown 2024-09-10T20:30:46.164 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:30:46.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient: get_monmap_and_config 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient: build_initial_monmap 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient: monmap: 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:46.287884+0000 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:46.287884+0000 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:46.290 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:46.291 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:46.291 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:46.291 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 1 Processor -- start 2024-09-10T20:30:46.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 1 -- start start 2024-09-10T20:30:46.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient: init 2024-09-10T20:30:46.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:46.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient: _add_conns ranks=[3,2,0,1,4] 2024-09-10T20:30:46.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdd500a41a0 0x7fdd500a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient(hunting): picked mon.noname-a con 0x7fdd500a41a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd500a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient(hunting): picked mon.noname-b con 0x7fdd500a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdd500a49a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd500a6a20 con 0x7fdd500a49a0 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fdd500a2570 con 0x7fdd500a41a0 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdd50070400 con 0x7fdd500a4580 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.284+0000 7fdd586f7640 10 monclient(hunting): authenticate will time out at 1206.427954s 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd500a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd56493640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fdd500a41a0 0x7fdd500a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd500a59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44754/0 (socket says 172.21.15.153:44754) 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 1 -- 172.21.15.153:0/323799632 learned_addr learned my addr 172.21.15.153:0/323799632 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd56c94640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fdd500a49a0 0x7fdd500a5f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44434/0 (socket says 172.21.15.153:44434) 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd56493640 10 monclient(hunting): get_auth_request con 0x7fdd500a41a0 auth_method 0 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd56493640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd56493640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd56493640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient(hunting): get_auth_request con 0x7fdd500a4580 auth_method 0 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd56493640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd56493640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd56493640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient(hunting): handle_auth_done global_id 24449 payload 274 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 1 -- 172.21.15.153:0/323799632 >> v2:172.21.15.153:3300/0 conn(0x7fdd500a41a0 msgr2=0x7fdd500a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 1 --2- 172.21.15.153:0/323799632 >> v2:172.21.15.153:3300/0 conn(0x7fdd500a41a0 0x7fdd500a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 1 -- 172.21.15.153:0/323799632 >> v1:172.21.15.153:6789/0 conn(0x7fdd500a49a0 legacy=0x7fdd500a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient: _finish_hunting 0 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 1 monclient: found mon.noname-b 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:30:46.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.288+0000 7fdd55c92640 1 -- 172.21.15.153:0/323799632 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd500a3c80 con 0x7fdd500a4580 2024-09-10T20:30:46.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 1 -- 172.21.15.153:0/323799632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 msgr2=0x7fdd500a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:46.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 1 --2- 172.21.15.153:0/323799632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd500a59d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009870 tx=0x7fdd4001c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e8d746ff77930554 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:46.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 1 -- 172.21.15.153:0/323799632 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd400b0070 con 0x7fdd500a4580 2024-09-10T20:30:46.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 10 monclient: dump: 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 10 monclient: _finish_auth 0 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 10 monclient: _check_auth_tickets 2024-09-10T20:30:46.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 1 -- 172.21.15.153:0/323799632 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd400ab600 con 0x7fdd500a4580 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 1 -- 172.21.15.153:0/323799632 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd400ab9d0 con 0x7fdd500a4580 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 5 monclient: authenticate success, global_id 24449 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 4 set_mon_vals no callback set 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55491640 10 monclient: dump: 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.300 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient: shutdown 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 -- 172.21.15.153:0/323799632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 msgr2=0x7fdd500a59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 --2- 172.21.15.153:0/323799632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd500a59d0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdd40009870 tx=0x7fdd4001c420 comp rx=0 tx=0).stop 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 -- 172.21.15.153:0/323799632 shutdown_connections 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 --2- 172.21.15.153:0/323799632 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd500a59d0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 --2- 172.21.15.153:0/323799632 >> v2:172.21.15.153:3300/0 conn(0x7fdd500a41a0 0x7fdd500a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 -- 172.21.15.153:0/323799632 >> 172.21.15.153:0/323799632 conn(0x7fdd500ab7e0 msgr2=0x7fdd500a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:46.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 -- 172.21.15.153:0/323799632 shutdown_connections 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 -- 172.21.15.153:0/323799632 wait complete. 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient: build_initial_monmap 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient: monmap: 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 librados: starting msgr at 2024-09-10T20:30:46.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 librados: starting objecter 2024-09-10T20:30:46.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 Processor -- start 2024-09-10T20:30:46.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 -- start start 2024-09-10T20:30:46.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 librados: setting wanted keys 2024-09-10T20:30:46.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 librados: calling monclient init 2024-09-10T20:30:46.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient: init 2024-09-10T20:30:46.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:46.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd500a41a0 0x7fdd50125b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient(hunting): picked mon.noname-a con 0x7fdd500a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd50123a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient(hunting): picked mon.noname-c con 0x7fdd500a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd500a49a0 0x7fdd5013c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient(hunting): picked mon.noname-b con 0x7fdd500a49a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fdd50143210 con 0x7fdd500a41a0 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fdd500a6830 con 0x7fdd500a49a0 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fdd50143390 con 0x7fdd500a4580 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd586f7640 10 monclient(hunting): authenticate will time out at 1206.435646s 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd50123a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd56493640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd500a41a0 0x7fdd50125b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd50123a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44770/0 (socket says 172.21.15.153:44770) 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 1 -- 172.21.15.153:0/3619623815 learned_addr learned my addr 172.21.15.153:0/3619623815 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 10 monclient(hunting): get_auth_request con 0x7fdd500a4580 auth_method 0 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd56493640 10 monclient(hunting): get_auth_request con 0x7fdd500a41a0 auth_method 0 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd56493640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd56493640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd56493640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd55c92640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd56493640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd56493640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.292+0000 7fdd56493640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 10 monclient(hunting): handle_auth_done global_id 34104 payload 824 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 1 -- 172.21.15.153:0/3619623815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 msgr2=0x7fdd50123a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 1 --2- 172.21.15.153:0/3619623815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd50123a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56c94640 1 --2- 172.21.15.153:0/3619623815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd500a49a0 0x7fdd5013c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 1 -- 172.21.15.153:0/3619623815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd500a49a0 msgr2=0x7fdd5013c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 1 --2- 172.21.15.153:0/3619623815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd500a49a0 0x7fdd5013c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 10 monclient: _finish_hunting 0 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 1 monclient: found mon.noname-a 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 1 -- 172.21.15.153:0/3619623815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd50124fa0 con 0x7fdd500a41a0 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56c94640 10 monclient: get_auth_request con 0x7fdd500a49a0 auth_method 0 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56c94640 1 --2- 172.21.15.153:0/3619623815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd500a49a0 0x7fdd5013c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 1 -- 172.21.15.153:0/3619623815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd500a41a0 msgr2=0x7fdd50125b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd56493640 1 --2- 172.21.15.153:0/3619623815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd500a41a0 0x7fdd50125b00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00e870 tx=0x7fdd4c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd46ffd640 10 client.?.objecter ms_handle_connect 0x7fdd500a41a0 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.296+0000 7fdd46ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 1 -- 172.21.15.153:0/3619623815 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c017070 con 0x7fdd500a41a0 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 monclient: dump: 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 monclient: _finish_auth 0 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:30:46.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 1 -- 172.21.15.153:0/3619623815 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd4c00cb50 con 0x7fdd500a41a0 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 5 monclient: authenticate success, global_id 34104 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 10 monclient: _renew_subs 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd500a21c0 con 0x7fdd500a41a0 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 1 -- 172.21.15.153:0/3619623815 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c013470 con 0x7fdd500a41a0 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 monclient: dump: 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.312 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 10 client.34104.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 10 monclient: _renew_subs 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd50126a60 con 0x7fdd500a41a0 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd54c90640 4 set_mon_vals no callback set 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 1 librados: init done 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:46.313 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 1 -- 172.21.15.153:0/3619623815 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fdd4c01a070 con 0x7fdd500a41a0 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd500a3720 con 0x7fdd500a41a0 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 1 --2- 172.21.15.153:0/3619623815 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdd3403cf20 0x7fdd3403f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 1 -- 172.21.15.153:0/3619623815 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(203..203 src has 1..203) ==== 5122+0+0 (secure 0 0 0) 0x7fdd4c04f880 con 0x7fdd500a41a0 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 10 client.34104.objecter ms_dispatch 0x7fdd50000ba0 osd_map(203..203 src has 1..203) 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 3 client.34104.objecter handle_osd_map got epochs [203,203] > 0 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 3 client.34104.objecter handle_osd_map decoding full epoch 203 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd46ffd640 20 client.34104.objecter dump_active .. 0 homeless 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd55c92640 1 --2- 172.21.15.153:0/3619623815 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdd3403cf20 0x7fdd3403f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.314 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.300+0000 7fdd55c92640 10 monclient: get_auth_request con 0x7fdd3403cf20 auth_method 0 2024-09-10T20:30:46.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.304+0000 7fdd55c92640 1 --2- 172.21.15.153:0/3619623815 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdd3403cf20 0x7fdd3403f3e0 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7fdd40000c10 tx=0x7fdd400a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:46.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.304+0000 7fdd46ffd640 10 client.34104.objecter ms_handle_connect 0x7fdd3403cf20 2024-09-10T20:30:46.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.304+0000 7fdd46ffd640 1 -- 172.21.15.153:0/3619623815 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fdd4c013e00 con 0x7fdd500a41a0 2024-09-10T20:30:46.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.304+0000 7fdd46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:46.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.304+0000 7fdd46ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:46.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.536+0000 7fdd586f7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:30:46.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.536+0000 7fdd586f7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:30:46.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.536+0000 7fdd586f7640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:46.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.536+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdd5006cd40 con 0x7fdd500a41a0 2024-09-10T20:30:46.542 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.536+0000 7fdd46ffd640 1 -- 172.21.15.153:0/3619623815 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fdd4c04e120 con 0x7fdd500a41a0 2024-09-10T20:30:46.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.536+0000 7fdd46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:30:46.543 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.536+0000 7fdd46ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:46.543 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:30:46.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 10 librados: watch_flush enter 2024-09-10T20:30:46.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 10 librados: watch_flush exit 2024-09-10T20:30:46.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 20 client.34104.objecter shutdown clearing up homeless session... 2024-09-10T20:30:46.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 10 client.34104.objecter successfully canceled tick 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdd3403cf20 msgr2=0x7fdd3403f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 --2- 172.21.15.153:0/3619623815 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdd3403cf20 0x7fdd3403f3e0 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7fdd40000c10 tx=0x7fdd400a6000 comp rx=0 tx=0).stop 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 10 monclient: shutdown 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd500a41a0 msgr2=0x7fdd50125b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 --2- 172.21.15.153:0/3619623815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd500a41a0 0x7fdd50125b00 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c00e870 tx=0x7fdd4c00ed30 comp rx=0 tx=0).stop 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 shutdown_connections 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 --2- 172.21.15.153:0/3619623815 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fdd500a49a0 0x7fdd5013c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 --2- 172.21.15.153:0/3619623815 >> v2:172.21.15.176:6812/3003473398 conn(0x7fdd3403cf20 0x7fdd3403f3e0 unknown :-1 s=CLOSED pgs=3315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 --2- 172.21.15.153:0/3619623815 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fdd500a4580 0x7fdd50123a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 --2- 172.21.15.153:0/3619623815 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fdd500a41a0 0x7fdd50125b00 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 >> 172.21.15.153:0/3619623815 conn(0x7fdd500ab7e0 msgr2=0x7fdd500a96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:46.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 shutdown_connections 2024-09-10T20:30:46.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 -- 172.21.15.153:0/3619623815 wait complete. 2024-09-10T20:30:46.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.540+0000 7fdd586f7640 1 librados: shutdown 2024-09-10T20:30:46.561 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T20:30:46.561 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T20:30:46.562 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:30:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient: get_monmap_and_config 2024-09-10T20:30:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient: build_initial_monmap 2024-09-10T20:30:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient: monmap: 2024-09-10T20:30:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:46.684106+0000 2024-09-10T20:30:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:46.684106+0000 2024-09-10T20:30:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:46.686 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:46.687 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:46.687 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 1 Processor -- start 2024-09-10T20:30:46.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 1 -- start start 2024-09-10T20:30:46.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient: init 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f67201364f0 0x7f67201368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient(hunting): picked mon.noname-c con 0x7f67201364f0 addr v2:172.21.15.176:3300/0 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f67201360a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f67200a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672009dc30 con 0x7f67201360a0 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67200a27a0 con 0x7f67200a1990 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f67200a2a80 con 0x7f67201364f0 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.680+0000 7f6724df2640 10 monclient(hunting): authenticate will time out at 1206.823831s 2024-09-10T20:30:46.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f67201360a0 0x7f672009ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44438/0 (socket says 172.21.15.153:44438) 2024-09-10T20:30:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 -- 172.21.15.153:0/1876361257 learned_addr learned my addr 172.21.15.153:0/1876361257 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:30:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 1 --2- 172.21.15.153:0/1876361257 >> v2:172.21.15.176:3300/0 conn(0x7f67201364f0 0x7f67201368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): get_auth_request con 0x7f67201364f0 auth_method 0 2024-09-10T20:30:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 -- 172.21.15.153:0/1876361257 >> v1:172.21.15.153:6789/0 conn(0x7f67201360a0 legacy=0x7f672009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1374088242 0 0) 0x7f672009dc30 con 0x7f67201360a0 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): _negotiate 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): my global_id is 34107 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6708003530 con 0x7f67201360a0 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 1 -- 172.21.15.153:0/1876361257 >> v1:172.21.15.176:6789/0 conn(0x7f67200a1990 legacy=0x7f672009d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4140417591 0 0) 0x7f6708003530 con 0x7f67201360a0 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 >> v2:172.21.15.176:3300/0 conn(0x7f67201364f0 msgr2=0x7f67201368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 --2- 172.21.15.153:0/1876361257 >> v2:172.21.15.176:3300/0 conn(0x7f67201364f0 0x7f67201368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 >> v1:172.21.15.176:6789/0 conn(0x7f67200a1990 legacy=0x7f672009d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient: _finish_hunting 0 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 monclient: found mon.noname-a-legacy 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67200a2d60 con 0x7f67201360a0 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient: _finish_auth 0 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient: _check_auth_tickets 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3765011595 0 0) 0x7f6714001d30 con 0x7f67201360a0 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f6724df2640 5 monclient: authenticate success, global_id 34107 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f6724df2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T20:30:46.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient: dump: 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.695 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 >> v1:172.21.15.153:6789/0 conn(0x7f67201360a0 legacy=0x7f672009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6708003b30 0x7f6708003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): picked mon.b con 0x7f6708003b30 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f6708004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): picked mon.a con 0x7f6708004810 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67080053d0 0x7f670800ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): picked mon.c con 0x7f67080053d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f67200a2a80 con 0x7f6708004810 2024-09-10T20:30:46.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f670800c310 con 0x7f6708003b30 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f670800c510 con 0x7f67080053d0 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671d59a640 10 monclient(hunting): _finish_auth 0 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f6724df2640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67080053d0 0x7f670800ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f6708004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient(hunting): get_auth_request con 0x7f6708004810 auth_method 0 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 10 monclient(hunting): get_auth_request con 0x7f67080053d0 auth_method 0 2024-09-10T20:30:46.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6708003b30 0x7f6708003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): get_auth_request con 0x7f6708003b30 auth_method 0 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient(hunting): handle_auth_done global_id 34107 payload 293 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 -- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67080053d0 msgr2=0x7f670800ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f67080053d0 0x7f670800ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 -- 172.21.15.153:0/1876361257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6708003b30 msgr2=0x7f6708003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f6708003b30 0x7f6708003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient: _finish_hunting 0 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 monclient: found mon.a 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 1 -- 172.21.15.153:0/1876361257 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6708003530 con 0x7f6708004810 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient: _finish_auth 0 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 10 monclient: _check_auth_tickets 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:46.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.684+0000 7f671ed9d640 1 -- 172.21.15.153:0/1876361257 reap_dead start 2024-09-10T20:30:46.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.688+0000 7f671dd9b640 1 -- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 msgr2=0x7f6708004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:46.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.688+0000 7f671dd9b640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f6708004c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6714001860 tx=0x7f671401d490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac230585ae002a7 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:46.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67140b1070 con 0x7f6708004810 2024-09-10T20:30:46.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:46.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 10 monclient: dump: 2024-09-10T20:30:46.703 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.703 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67140a8070 con 0x7f6708004810 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:46.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 4 set_mon_vals no callback set 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 1 -- 172.21.15.153:0/1876361257 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671400a690 con 0x7f6708004810 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671d59a640 10 monclient: dump: 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.705 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient: shutdown 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 -- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 msgr2=0x7f6708004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f6708004c90 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6714001860 tx=0x7f671401d490 comp rx=0 tx=0).stop 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 -- 172.21.15.153:0/1876361257 shutdown_connections 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 --2- 172.21.15.153:0/1876361257 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f6708004c90 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 -- 172.21.15.153:0/1876361257 >> 172.21.15.153:0/1876361257 conn(0x7f67200a71b0 msgr2=0x7f67200a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 -- 172.21.15.153:0/1876361257 shutdown_connections 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 -- 172.21.15.153:0/1876361257 wait complete. 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient: build_initial_monmap 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient: monmap: 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:46.706 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 librados: starting msgr at 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 librados: starting objecter 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 Processor -- start 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 -- start start 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 librados: setting wanted keys 2024-09-10T20:30:46.707 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 librados: calling monclient init 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient: init 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f6720073bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient(hunting): picked mon.noname-a con 0x7f6708004810 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f672009e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient(hunting): picked mon.noname-c con 0x7f672009e2e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f67201d75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient(hunting): picked mon.noname-b con 0x7f672009ec80 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f672009d9b0 con 0x7f6708004810 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f672009d830 con 0x7f672009ec80 2024-09-10T20:30:46.708 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f67200a08a0 con 0x7f672009e2e0 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f6724df2640 10 monclient(hunting): authenticate will time out at 1206.834461s 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f672009e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f6720073bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f672009e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44788/0 (socket says 172.21.15.153:44788) 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 learned_addr learned my addr 172.21.15.153:0/3046582782 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient(hunting): get_auth_request con 0x7f672009e2e0 auth_method 0 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.709 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671e59c640 10 monclient(hunting): get_auth_request con 0x7f6708004810 auth_method 0 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671ed9d640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f67201d75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671ed9d640 10 monclient(hunting): get_auth_request con 0x7f672009ec80 auth_method 0 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.710 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient(hunting): handle_auth_done global_id 24455 payload 824 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 msgr2=0x7f67201d75b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f67201d75b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 msgr2=0x7f6720073bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f6720073bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient: _finish_hunting 0 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 monclient: found mon.noname-c 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67200721d0 con 0x7f672009e2e0 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671e59c640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f6720073bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 msgr2=0x7f672009e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 msgr2=0x7f672009e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:30:46.711 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f672009e740 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f67140af040 tx=0x7f6708005840 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:30:46.712 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.692+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f672009e740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:30:46.902 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f672009e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient: get_auth_request con 0x7f672009e2e0 auth_method 0 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 0 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f672009e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f672009e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 msgr2=0x7f672009e740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f672009e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f67000054c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient(hunting): picked mon.noname-a con 0x7f6708004810 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 0x7f6700008390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient(hunting): picked mon.noname-c con 0x7f6700005f00 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:46.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f670000af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient(hunting): picked mon.noname-b con 0x7f672009ec80 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f670000b960 con 0x7f6708004810 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f670000bb10 con 0x7f672009ec80 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f670000bd10 con 0x7f6700005f00 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f670effd640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 0x7f6700008390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient(hunting): get_auth_request con 0x7f6700005f00 auth_method 0 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:46.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671e59c640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f67000054c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671ed9d640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f670000af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671e59c640 10 monclient(hunting): get_auth_request con 0x7f6708004810 auth_method 0 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671ed9d640 10 monclient(hunting): get_auth_request con 0x7f672009ec80 auth_method 0 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:46.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient(hunting): handle_auth_done global_id 24455 payload 293 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 msgr2=0x7f670000af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f670000af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 msgr2=0x7f67000054c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f67000054c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient: _finish_hunting 0 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 1 monclient: found mon.noname-c 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f670000bfd0 con 0x7f6700005f00 2024-09-10T20:30:46.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.896+0000 7f671ed9d640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f670000af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:30:46.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 msgr2=0x7f6700008390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:46.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 0x7f6700008390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f67200726c0 tx=0x7f67200a2970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:46.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 client.?.objecter ms_handle_connect 0x7f6700005f00 2024-09-10T20:30:46.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:46.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67140b1040 con 0x7f6700005f00 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: dump: 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: _finish_auth 0 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: _check_auth_tickets 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67140bc070 con 0x7f6700005f00 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 5 monclient: authenticate success, global_id 24455 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 10 monclient: _renew_subs 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67200724b0 con 0x7f6700005f00 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67140aca90 con 0x7f6700005f00 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: dump: 2024-09-10T20:30:46.913 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671cd99640 4 set_mon_vals no callback set 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 10 client.24455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 10 monclient: _renew_subs 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:46.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f6720071ef0 con 0x7f6700005f00 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 1 librados: init done 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 msgr2=0x7f6700008390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 msgr2=0x7f6700008390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 0x7f6700008390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f67200726c0 tx=0x7f67200a2970 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 0x7f6700008390 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f67200726c0 tx=0x7f67200a2970 comp rx=0 tx=0).stop 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: ms_handle_reset current mon [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 msgr2=0x7f6700008390 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6700005f00 0x7f6700008390 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f67000054c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient(hunting): picked mon.a con 0x7f6708004810 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:46.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f670000fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient(hunting): picked mon.c con 0x7f672009e2e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f670000af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient(hunting): picked mon.b con 0x7f672009ec80 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6700010b10 con 0x7f6708004810 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f6700010c90 con 0x7f672009ec80 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f6700010e10 con 0x7f672009e2e0 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f670effd640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f672009e6c0 0x7f6720073bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f6720070cb0 con 0x7f672009e6c0 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f67000054c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): get_auth_request con 0x7f6708004810 auth_method 0 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f670000fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f672009e6c0 0x7f6720073bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): get_auth_request con 0x7f672009e6c0 auth_method 0 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 10 monclient(hunting): get_auth_request con 0x7f672009e2e0 auth_method 0 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671ed9d640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f670000af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671ed9d640 10 monclient(hunting): get_auth_request con 0x7f672009ec80 auth_method 0 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671ed9d640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): handle_auth_done global_id 24455 payload 293 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 msgr2=0x7f670000fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f672009e2e0 0x7f670000fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 msgr2=0x7f670000af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f672009ec80 0x7f670000af50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient: _finish_hunting 0 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 1 monclient: found mon.a 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:46.919 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f6700010fc0 con 0x7f6708004810 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient: _finish_auth 0 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient: _check_auth_tickets 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 10 monclient(hunting): handle_auth_done global_id 34113 payload 824 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.900+0000 7f671e59c640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f672009eaa0 con 0x7f672009e6c0 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f671e59c640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 msgr2=0x7f67000054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 client.24455.objecter ms_handle_connect 0x7f6708004810 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 client.24455.objecter resend_mon_ops 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f671e59c640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f67000054c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f67100090e0 tx=0x7f67140a8000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f671e59c640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f672009e6c0 msgr2=0x7f6720073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 client.24455.objecter ms_handle_connect 0x7f672009e6c0 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f671e59c640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f672009e6c0 0x7f6720073bf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6710014800 tx=0x7f6710014cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:46.920 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 client.24455.objecter resend_mon_ops 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6710028070 con 0x7f6708004810 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 monclient: dump: 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.921 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6710032070 con 0x7f6708004810 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670ffff640 4 set_mon_vals no callback set 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67100186c0 con 0x7f672009e6c0 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 monclient: dump: 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.922 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f671002b070 con 0x7f6708004810 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6700005f00 0x7f6700048110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6710052570 con 0x7f672009e6c0 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f672009e6c0 msgr2=0x7f6720073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f672009e6c0 0x7f6720073bf0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6710014800 tx=0x7f6710014cc0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f672009e6c0 msgr2=0x7f6720073bf0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f672009e6c0 0x7f6720073bf0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6710014800 tx=0x7f6710014cc0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.0 v2:172.21.15.153:3300/0 4 ==== osd_map(204..204 src has 1..204) ==== 4901+0+0 (secure 0 0 0) 0x7f67100521f0 con 0x7f6708004810 2024-09-10T20:30:46.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 10 client.24455.objecter ms_dispatch 0x7f6720000ba0 osd_map(204..204 src has 1..204) 2024-09-10T20:30:46.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 3 client.24455.objecter handle_osd_map got epochs [204,204] > 0 2024-09-10T20:30:46.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 3 client.24455.objecter handle_osd_map decoding full epoch 204 2024-09-10T20:30:46.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f670effd640 20 client.24455.objecter dump_active .. 0 homeless 2024-09-10T20:30:46.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6700005f00 0x7f6700048110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.904+0000 7f671dd9b640 10 monclient: get_auth_request con 0x7f6700005f00 auth_method 0 2024-09-10T20:30:46.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.908+0000 7f671dd9b640 1 --2- 172.21.15.153:0/3046582782 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6700005f00 0x7f6700048110 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f671401dd30 tx=0x7f6714007050 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.908+0000 7f670effd640 10 client.24455.objecter ms_handle_connect 0x7f6700005f00 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f6724df2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f6724df2640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67201c0250 0x7f67201c0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f6724df2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f672009dbc0 con 0x7f67201c0250 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67201c0250 0x7f67201c0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 10 monclient: get_auth_request con 0x7f67201c0250 auth_method 0 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:46.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 10 monclient(hunting): handle_auth_done global_id 34116 payload 824 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.912+0000 7f671ed9d640 1 -- 172.21.15.153:0/3046582782 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f672009e0a0 con 0x7f67201c0250 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 10 client.24455.objecter ms_handle_connect 0x7f67201c0250 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 10 client.24455.objecter resend_mon_ops 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f671ed9d640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67201c0250 msgr2=0x7f67201c0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f671ed9d640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67201c0250 0x7f67201c0660 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f672009de60 tx=0x7f6718015bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6718015e60 con 0x7f67201c0250 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 10 monclient: dump: 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:46.927 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f671801a030 con 0x7f67201c0250 2024-09-10T20:30:46.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:30:46.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:46.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67201c0250 msgr2=0x7f67201c0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67201c0250 0x7f67201c0660 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f672009de60 tx=0x7f6718015bb0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67201c0250 msgr2=0x7f67201c0660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:46.929 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f670effd640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f67201c0250 0x7f67201c0660 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f672009de60 tx=0x7f6718015bb0 comp rx=0 tx=0).stop 2024-09-10T20:30:46.929 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:30:46.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:30:46.929 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:30:46.929 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 26, 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 7475, 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:46.930 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:46.931 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:30:46.932 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:46.933 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:46.934 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.935 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.936 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:30:46.937 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:46.938 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:46.939 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:30:46.940 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:46.941 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:46.942 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:30:46.942 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:30:46.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 10 librados: watch_flush enter 2024-09-10T20:30:46.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 10 librados: watch_flush exit 2024-09-10T20:30:46.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 20 client.24455.objecter shutdown clearing up homeless session... 2024-09-10T20:30:46.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 10 client.24455.objecter successfully canceled tick 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6700005f00 msgr2=0x7f6700048110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 1 --2- 172.21.15.153:0/3046582782 >> v2:172.21.15.176:6812/3003473398 conn(0x7f6700005f00 0x7f6700048110 secure :-1 s=READY pgs=3316 cs=0 l=1 rev1=1 crypto rx=0x7f671401dd30 tx=0x7f6714007050 comp rx=0 tx=0).stop 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 10 monclient: shutdown 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 msgr2=0x7f67000054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 1 --2- 172.21.15.153:0/3046582782 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f6708004810 0x7f67000054c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f67100090e0 tx=0x7f67140a8000 comp rx=0 tx=0).stop 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f671ed9d640 1 -- 172.21.15.153:0/3046582782 reap_dead start 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f671ed9d640 1 -- 172.21.15.153:0/3046582782 reap_dead start 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 shutdown_connections 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 >> 172.21.15.153:0/3046582782 conn(0x7f67200a71b0 msgr2=0x7f67200a5210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 shutdown_connections 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 1 -- 172.21.15.153:0/3046582782 wait complete. 2024-09-10T20:30:46.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:46.916+0000 7f6724df2640 1 librados: shutdown 2024-09-10T20:30:46.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:30:46.945 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T20:30:46.945 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient: get_monmap_and_config 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient: build_initial_monmap 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient: monmap: 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:47.069976+0000 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:47.069976+0000 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:47.072 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:47.073 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:47.073 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:47.073 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 1 Processor -- start 2024-09-10T20:30:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 1 -- start start 2024-09-10T20:30:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient: init 2024-09-10T20:30:47.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient: _add_conns ranks=[4,3,2,0,1] 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe8781364f0 0x7fe8781368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient(hunting): picked mon.noname-c con 0x7fe8781364f0 addr v2:172.21.15.176:3300/0 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8781360a0 0x7fe87809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient(hunting): picked mon.noname-a con 0x7fe8781360a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 0x7fe87809d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient(hunting): picked mon.noname-b con 0x7fe8780a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fe8780a27a0 con 0x7fe8781360a0 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe8780a2a80 con 0x7fe8781364f0 2024-09-10T20:30:47.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe8780a2d60 con 0x7fe8780a1990 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87fea6640 10 monclient(hunting): authenticate will time out at 1207.209626s 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8781360a0 0x7fe87809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 0x7fe87809d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8781360a0 0x7fe87809ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49724/0 (socket says 172.21.15.153:49724) 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 1 -- 172.21.15.153:0/3707846011 learned_addr learned my addr 172.21.15.153:0/3707846011 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 10 monclient(hunting): get_auth_request con 0x7fe8781360a0 auth_method 0 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient(hunting): get_auth_request con 0x7fe8780a1990 auth_method 0 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87dc42640 1 --2- 172.21.15.153:0/3707846011 >> v2:172.21.15.176:3300/0 conn(0x7fe8781364f0 0x7fe8781368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87dc42640 10 monclient(hunting): get_auth_request con 0x7fe8781364f0 auth_method 0 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87dc42640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient(hunting): handle_auth_done global_id 24458 payload 274 2024-09-10T20:30:47.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 1 -- 172.21.15.153:0/3707846011 >> v2:172.21.15.176:3300/0 conn(0x7fe8781364f0 msgr2=0x7fe8781368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 1 --2- 172.21.15.153:0/3707846011 >> v2:172.21.15.176:3300/0 conn(0x7fe8781364f0 0x7fe8781368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 1 -- 172.21.15.153:0/3707846011 >> v2:172.21.15.153:3300/0 conn(0x7fe8781360a0 msgr2=0x7fe87809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 1 --2- 172.21.15.153:0/3707846011 >> v2:172.21.15.153:3300/0 conn(0x7fe8781360a0 0x7fe87809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient: _finish_hunting 0 2024-09-10T20:30:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 1 monclient: found mon.noname-b 2024-09-10T20:30:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:30:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87e443640 1 -- 172.21.15.153:0/3707846011 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe878136e10 con 0x7fe8780a1990 2024-09-10T20:30:47.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.068+0000 7fe87d441640 1 --2- 172.21.15.153:0/3707846011 >> v2:172.21.15.153:3300/0 conn(0x7fe8781360a0 0x7fe87809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87e443640 1 -- 172.21.15.153:0/3707846011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 msgr2=0x7fe87809d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87e443640 1 --2- 172.21.15.153:0/3707846011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 0x7fe87809d350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe8780a12f0 tx=0x7fe87001c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfda8516c7480fa2 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 1 -- 172.21.15.153:0/3707846011 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8700b2070 con 0x7fe8780a1990 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 10 monclient: dump: 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.086 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 10 monclient: _finish_auth 0 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 10 monclient: _check_auth_tickets 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 1 -- 172.21.15.153:0/3707846011 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8700a5780 con 0x7fe8780a1990 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 1 -- 172.21.15.153:0/3707846011 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8700a5b50 con 0x7fe8780a1990 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 5 monclient: authenticate success, global_id 24458 2024-09-10T20:30:47.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 4 set_mon_vals no callback set 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87cc40640 10 monclient: dump: 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.088 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 10 monclient: shutdown 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 -- 172.21.15.153:0/3707846011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 msgr2=0x7fe87809d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 --2- 172.21.15.153:0/3707846011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 0x7fe87809d350 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe8780a12f0 tx=0x7fe87001c7f0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 -- 172.21.15.153:0/3707846011 shutdown_connections 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 --2- 172.21.15.153:0/3707846011 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 0x7fe87809d350 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 --2- 172.21.15.153:0/3707846011 >> v2:172.21.15.153:3300/0 conn(0x7fe8781360a0 0x7fe87809ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 --2- 172.21.15.153:0/3707846011 >> v2:172.21.15.176:3300/0 conn(0x7fe8781364f0 0x7fe8781368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 -- 172.21.15.153:0/3707846011 >> 172.21.15.153:0/3707846011 conn(0x7fe8780a71b0 msgr2=0x7fe8780a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 -- 172.21.15.153:0/3707846011 shutdown_connections 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 -- 172.21.15.153:0/3707846011 wait complete. 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 10 monclient: build_initial_monmap 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 10 monclient: monmap: 2024-09-10T20:30:47.089 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 librados: starting msgr at 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.072+0000 7fe87fea6640 1 librados: starting objecter 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 Processor -- start 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 -- start start 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 librados: setting wanted keys 2024-09-10T20:30:47.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 librados: calling monclient init 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient: init 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 0x7fe8781d0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient(hunting): picked mon.noname-c con 0x7fe8780a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781360a0 0x7fe87809f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient(hunting): picked mon.noname-a con 0x7fe8781360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8781364f0 0x7fe87809fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient(hunting): picked mon.noname-b con 0x7fe8781364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe87813ffe0 con 0x7fe8781360a0 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe87813fe60 con 0x7fe8781364f0 2024-09-10T20:30:47.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe878140160 con 0x7fe8780a1990 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient(hunting): authenticate will time out at 1207.216712s 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87dc42640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 0x7fe8781d0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781360a0 0x7fe87809f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781360a0 0x7fe87809f810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49730/0 (socket says 172.21.15.153:49730) 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87dc42640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 0x7fe8781d0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44848/0 (socket says 172.21.15.153:44848) 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 -- 172.21.15.153:0/3074981620 learned_addr learned my addr 172.21.15.153:0/3074981620 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient(hunting): get_auth_request con 0x7fe8781360a0 auth_method 0 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87dc42640 10 monclient(hunting): get_auth_request con 0x7fe8780a1990 auth_method 0 2024-09-10T20:30:47.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87dc42640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87dc42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87dc42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87dc42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87e443640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8781364f0 0x7fe87809fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient(hunting): handle_auth_done global_id 34122 payload 824 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 msgr2=0x7fe8781d0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8780a1990 0x7fe8781d0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8781364f0 msgr2=0x7fe87809fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8781364f0 0x7fe87809fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient: _finish_hunting 0 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 monclient: found mon.noname-a 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 -- 172.21.15.153:0/3074981620 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe878074400 con 0x7fe8781360a0 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87e443640 10 monclient: get_auth_request con 0x7fe8781364f0 auth_method 0 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87e443640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8781364f0 0x7fe87809fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781360a0 msgr2=0x7fe87809f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87d441640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781360a0 0x7fe87809f810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe86800e9d0 tx=0x7fe86800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 client.?.objecter ms_handle_connect 0x7fe8781360a0 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868017070 con 0x7fe8781360a0 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 monclient: dump: 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 monclient: _finish_auth 0 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:30:47.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 5 monclient: authenticate success, global_id 34122 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient: _renew_subs 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86800cc70 con 0x7fe8781360a0 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe878131860 con 0x7fe8781360a0 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe868013670 con 0x7fe8781360a0 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86e7fc640 10 monclient: dump: 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.096 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe86ffff640 4 set_mon_vals no callback set 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 client.34122.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient: _renew_subs 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe878074120 con 0x7fe8781360a0 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.076+0000 7fe87fea6640 1 librados: init done 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87fea6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:47.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87fea6640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813b610 0x7fe87813b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87fea6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe87809e0a0 con 0x7fe87813b610 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813b610 0x7fe87813b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 10 monclient: get_auth_request con 0x7fe87813b610 auth_method 0 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fe86801a070 con 0x7fe8781360a0 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 10 monclient(hunting): handle_auth_done global_id 24464 payload 824 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 1 -- 172.21.15.153:0/3074981620 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe8781368d0 con 0x7fe87813b610 2024-09-10T20:30:47.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 1 --2- 172.21.15.153:0/3074981620 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe85c03cea0 0x7fe85c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 4901+0+0 (secure 0 0 0) 0x7fe86804f770 con 0x7fe8781360a0 2024-09-10T20:30:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 10 client.34122.objecter ms_dispatch 0x7fe878000ba0 osd_map(204..204 src has 1..204) 2024-09-10T20:30:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 3 client.34122.objecter handle_osd_map got epochs [204,204] > 0 2024-09-10T20:30:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 3 client.34122.objecter handle_osd_map decoding full epoch 204 2024-09-10T20:30:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 20 client.34122.objecter dump_active .. 0 homeless 2024-09-10T20:30:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87e443640 1 --2- 172.21.15.153:0/3074981620 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe85c03cea0 0x7fe85c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87e443640 10 monclient: get_auth_request con 0x7fe85c03cea0 auth_method 0 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 10 client.34122.objecter ms_handle_connect 0x7fe87813b610 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 10 client.34122.objecter resend_mon_ops 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813b610 msgr2=0x7fe87813b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe87dc42640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813b610 0x7fe87813b9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe874006aa0 tx=0x7fe874010c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874018020 con 0x7fe87813b610 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 10 monclient: dump: 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.102 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe874004450 con 0x7fe87813b610 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813b610 msgr2=0x7fe87813b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813b610 0x7fe87813b9f0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe874006aa0 tx=0x7fe874010c80 comp rx=0 tx=0).stop 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813b610 msgr2=0x7fe87813b9f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.080+0000 7fe86e7fc640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813b610 0x7fe87813b9f0 secure :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe874006aa0 tx=0x7fe874010c80 comp rx=0 tx=0).stop 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.084+0000 7fe87e443640 1 --2- 172.21.15.153:0/3074981620 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe85c03cea0 0x7fe85c03f360 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7fe8780a13d0 tx=0x7fe8700a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.084+0000 7fe86e7fc640 10 client.34122.objecter ms_handle_connect 0x7fe85c03cea0 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87fea6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87fea6640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813bf30 0x7fe87812e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87fea6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe878072f00 con 0x7fe87813bf30 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813bf30 0x7fe87812e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 10 monclient: get_auth_request con 0x7fe87813bf30 auth_method 0 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 10 monclient(hunting): handle_auth_done global_id 24467 payload 824 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.088+0000 7fe87dc42640 1 -- 172.21.15.153:0/3074981620 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe87806a660 con 0x7fe87813bf30 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 10 client.34122.objecter ms_handle_connect 0x7fe87813bf30 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 10 client.34122.objecter resend_mon_ops 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87dc42640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813bf30 msgr2=0x7fe87812e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87dc42640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813bf30 0x7fe87812e720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe874006aa0 tx=0x7fe8740046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874018030 con 0x7fe87813bf30 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:47.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 10 monclient: dump: 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7fe87400ecb0 con 0x7fe87813bf30 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:47.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813bf30 msgr2=0x7fe87812e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813bf30 0x7fe87812e720 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe874006aa0 tx=0x7fe8740046d0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813bf30 msgr2=0x7fe87812e720 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:47.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe86e7fc640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe87813bf30 0x7fe87812e720 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe874006aa0 tx=0x7fe8740046d0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.107 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:30:47.107 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:30:47.107 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:30:47.107 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:30:47.107 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 26, 2024-09-10T20:30:47.107 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 148661, 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:47.108 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:47.109 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.110 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:30:47.111 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:47.112 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:47.113 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:30:47.114 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.115 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.116 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:47.118 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:47.119 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.120 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.120 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:30:47.120 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:30:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 10 librados: watch_flush enter 2024-09-10T20:30:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 10 librados: watch_flush exit 2024-09-10T20:30:47.121 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 20 client.34122.objecter shutdown clearing up homeless session... 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 10 client.34122.objecter successfully canceled tick 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe85c03cea0 msgr2=0x7fe85c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 1 --2- 172.21.15.153:0/3074981620 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe85c03cea0 0x7fe85c03f360 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7fe8780a13d0 tx=0x7fe8700a7070 comp rx=0 tx=0).stop 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 10 monclient: shutdown 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781360a0 msgr2=0x7fe87809f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 1 --2- 172.21.15.153:0/3074981620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8781360a0 0x7fe87809f810 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe86800e9d0 tx=0x7fe86800ee90 comp rx=0 tx=0).stop 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87e443640 1 -- 172.21.15.153:0/3074981620 reap_dead start 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87e443640 1 -- 172.21.15.153:0/3074981620 reap_dead start 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 shutdown_connections 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 >> 172.21.15.153:0/3074981620 conn(0x7fe8780a71b0 msgr2=0x7fe878132940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 shutdown_connections 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 1 -- 172.21.15.153:0/3074981620 wait complete. 2024-09-10T20:30:47.122 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.092+0000 7fe87fea6640 1 librados: shutdown 2024-09-10T20:30:47.123 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:30:47.123 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T20:30:47.123 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:30:47.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient: get_monmap_and_config 2024-09-10T20:30:47.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient: build_initial_monmap 2024-09-10T20:30:47.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient: monmap: 2024-09-10T20:30:47.233 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:47.233 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.233 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:47.231351+0000 2024-09-10T20:30:47.233 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:47.231351+0000 2024-09-10T20:30:47.234 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:47.234 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.234 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:47.234 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:47.234 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:47.234 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:47.234 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:47.234 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 1 Processor -- start 2024-09-10T20:30:47.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 1 -- start start 2024-09-10T20:30:47.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient: init 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient: _add_conns ranks=[1,2,0,3,4] 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f53080a5ab0 addr v1:172.21.15.176:6789/0 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a65a0 0x7f530813dc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient(hunting): picked mon.noname-b con 0x7f53080a65a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f530813e1b0 addr v1:172.21.15.153:6789/0 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5308070780 con 0x7f530813e1b0 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53080709c0 con 0x7f53080a5ab0 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f53080a3010 con 0x7f53080a65a0 2024-09-10T20:30:47.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f5310280640 10 monclient(hunting): authenticate will time out at 1207.371151s 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a65a0 0x7f530813dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530e81d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f530813e1b0 0x7f53081405a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44444/0 (socket says 172.21.15.153:44444) 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient(hunting): get_auth_request con 0x7f53080a65a0 auth_method 0 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530e81d640 1 -- 172.21.15.153:0/961100226 learned_addr learned my addr 172.21.15.153:0/961100226 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient(hunting): handle_auth_done global_id 24470 payload 274 2024-09-10T20:30:47.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 1 -- 172.21.15.153:0/961100226 >> v1:172.21.15.176:6789/0 conn(0x7f53080a5ab0 legacy=0x7f53080a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:47.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 1 -- 172.21.15.153:0/961100226 >> v1:172.21.15.153:6789/0 conn(0x7f530813e1b0 legacy=0x7f53081405a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:47.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient: _finish_hunting 0 2024-09-10T20:30:47.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 1 monclient: found mon.noname-b 2024-09-10T20:30:47.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:47.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:30:47.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.228+0000 7f530d81b640 1 -- 172.21.15.153:0/961100226 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53080a26e0 con 0x7f53080a65a0 2024-09-10T20:30:47.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.232+0000 7f530d81b640 1 -- 172.21.15.153:0/961100226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a65a0 msgr2=0x7f530813dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.232+0000 7f530d81b640 1 --2- 172.21.15.153:0/961100226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a65a0 0x7f530813dc70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f52f80097f0 tx=0x7f52f801c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23d98e82b0d0a14b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 1 -- 172.21.15.153:0/961100226 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f80b0070 con 0x7f53080a65a0 2024-09-10T20:30:47.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:30:47.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 10 monclient: dump: 2024-09-10T20:30:47.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.245 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.245 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.245 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.245 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.245 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.245 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.245 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 10 monclient: _finish_auth 0 2024-09-10T20:30:47.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 10 monclient: _check_auth_tickets 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 1 -- 172.21.15.153:0/961100226 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52f80ab600 con 0x7f53080a65a0 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 5 monclient: authenticate success, global_id 24470 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 4 set_mon_vals no callback set 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 1 -- 172.21.15.153:0/961100226 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52f80ab9d0 con 0x7f53080a65a0 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d01a640 10 monclient: dump: 2024-09-10T20:30:47.246 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient: shutdown 2024-09-10T20:30:47.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 -- 172.21.15.153:0/961100226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a65a0 msgr2=0x7f530813dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 --2- 172.21.15.153:0/961100226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a65a0 0x7f530813dc70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f52f80097f0 tx=0x7f52f801c3a0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 -- 172.21.15.153:0/961100226 shutdown_connections 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 --2- 172.21.15.153:0/961100226 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a65a0 0x7f530813dc70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 -- 172.21.15.153:0/961100226 >> 172.21.15.153:0/961100226 conn(0x7f53080aba00 msgr2=0x7f53080a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 -- 172.21.15.153:0/961100226 shutdown_connections 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 -- 172.21.15.153:0/961100226 wait complete. 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient: build_initial_monmap 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient: monmap: 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:47.248 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:47.249 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:47.249 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:47.249 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:47.249 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.249 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:47.249 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:47.249 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:47.249 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 librados: starting msgr at 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 librados: starting objecter 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 Processor -- start 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 -- start start 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 librados: setting wanted keys 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 librados: calling monclient init 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient: init 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:30:47.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a5ab0 0x7f53081381b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient(hunting): picked mon.noname-c con 0x7f53080a5ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53080a65a0 0x7f530809e200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient(hunting): picked mon.noname-b con 0x7f53080a65a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f530813e1b0 0x7f530809e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient(hunting): picked mon.noname-a con 0x7f530813e1b0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f53080a3b70 con 0x7f530813e1b0 2024-09-10T20:30:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f53081444e0 con 0x7f53080a65a0 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f53080a0a00 con 0x7f53080a5ab0 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f5310280640 10 monclient(hunting): authenticate will time out at 1207.378366s 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f530813e1b0 0x7f530809e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e01c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a5ab0 0x7f53081381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f530813e1b0 0x7f530809e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49734/0 (socket says 172.21.15.153:49734) 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 1 -- 172.21.15.153:0/2470513821 learned_addr learned my addr 172.21.15.153:0/2470513821 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient(hunting): get_auth_request con 0x7f530813e1b0 auth_method 0 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e01c640 10 monclient(hunting): get_auth_request con 0x7f53080a5ab0 auth_method 0 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e01c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e01c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e01c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e01c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e01c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e01c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d81b640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53080a65a0 0x7f530809e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d81b640 10 monclient(hunting): get_auth_request con 0x7f53080a65a0 auth_method 0 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d81b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d81b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530d81b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient(hunting): handle_auth_done global_id 34125 payload 824 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a5ab0 msgr2=0x7f53081381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f53080a5ab0 0x7f53081381b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53080a65a0 msgr2=0x7f530809e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53080a65a0 0x7f530809e200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient: _finish_hunting 0 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 1 monclient: found mon.noname-a 2024-09-10T20:30:47.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:47.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.236+0000 7f530e81d640 1 -- 172.21.15.153:0/2470513821 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53080a22a0 con 0x7f530813e1b0 2024-09-10T20:30:47.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e81d640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f530813e1b0 msgr2=0x7f530809e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e81d640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f530813e1b0 0x7f530809e740 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f530000caf0 tx=0x7f530000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 client.?.objecter ms_handle_connect 0x7f530813e1b0 2024-09-10T20:30:47.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:47.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5300012400 con 0x7f530813e1b0 2024-09-10T20:30:47.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 monclient: dump: 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.260 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 monclient: _finish_auth 0 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 monclient: _check_auth_tickets 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53000125c0 con 0x7f530813e1b0 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 5 monclient: authenticate success, global_id 34125 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 10 monclient: _renew_subs 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5308121950 con 0x7f530813e1b0 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53000129b0 con 0x7f530813e1b0 2024-09-10T20:30:47.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 monclient: dump: 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.262 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530c819640 4 set_mon_vals no callback set 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 10 client.34125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 10 monclient: _renew_subs 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53080a0110 con 0x7f530813e1b0 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 1 librados: init done 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53081386f0 0x7f530809ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f53080a20e0 con 0x7f53081386f0 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f5300012bd0 con 0x7f530813e1b0 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53081386f0 0x7f530809ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 1 --2- 172.21.15.153:0/2470513821 >> v2:172.21.15.176:6812/3003473398 conn(0x7f52ec03cea0 0x7f52ec03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 10 monclient: get_auth_request con 0x7f53081386f0 auth_method 0 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 4901+0+0 (secure 0 0 0) 0x7f530004fe80 con 0x7f530813e1b0 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 10 client.34125.objecter ms_dispatch 0x7f5308000ba0 osd_map(204..204 src has 1..204) 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 3 client.34125.objecter handle_osd_map got epochs [204,204] > 0 2024-09-10T20:30:47.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 3 client.34125.objecter handle_osd_map decoding full epoch 204 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f52feffd640 20 client.34125.objecter dump_active .. 0 homeless 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530d81b640 1 --2- 172.21.15.153:0/2470513821 >> v2:172.21.15.176:6812/3003473398 conn(0x7f52ec03cea0 0x7f52ec03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530d81b640 10 monclient: get_auth_request con 0x7f52ec03cea0 auth_method 0 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 10 monclient(hunting): handle_auth_done global_id 19996 payload 824 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.240+0000 7f530e01c640 1 -- 172.21.15.153:0/2470513821 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f53080a5e90 con 0x7f53081386f0 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 10 client.34125.objecter ms_handle_connect 0x7f53081386f0 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f530e01c640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53081386f0 msgr2=0x7f530809ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 10 client.34125.objecter resend_mon_ops 2024-09-10T20:30:47.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f530e01c640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53081386f0 0x7f530809ec80 secure :-1 s=READY pgs=5641 cs=0 l=1 rev1=1 crypto rx=0x7f530400f770 tx=0x7f530400fc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 10 client.34125.objecter ms_handle_connect 0x7f52ec03cea0 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f530d81b640 1 --2- 172.21.15.153:0/2470513821 >> v2:172.21.15.176:6812/3003473398 conn(0x7f52ec03cea0 0x7f52ec03f360 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f530806a960 tx=0x7f52f80a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530400fe40 con 0x7f53081386f0 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 10 monclient: dump: 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.266 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5304019070 con 0x7f53081386f0 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53081386f0 msgr2=0x7f530809ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53081386f0 0x7f530809ec80 secure :-1 s=READY pgs=5641 cs=0 l=1 rev1=1 crypto rx=0x7f530400f770 tx=0x7f530400fc30 comp rx=0 tx=0).stop 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53081386f0 msgr2=0x7f530809ec80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.244+0000 7f52feffd640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f53081386f0 0x7f530809ec80 secure :-1 s=CLOSED pgs=5641 cs=0 l=1 rev1=1 crypto rx=0x7f530400f770 tx=0x7f530400fc30 comp rx=0 tx=0).stop 2024-09-10T20:30:47.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f5310280640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f5310280640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f530809f1c0 0x7f530813bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f5310280640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f530809f5d0 con 0x7f530809f1c0 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f530809f1c0 0x7f530813bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 10 monclient: get_auth_request con 0x7f530809f1c0 auth_method 0 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 10 monclient(hunting): handle_auth_done global_id 19999 payload 824 2024-09-10T20:30:47.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.252+0000 7f530e01c640 1 -- 172.21.15.153:0/2470513821 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f53080a13a0 con 0x7f530809f1c0 2024-09-10T20:30:47.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f530e01c640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f530809f1c0 msgr2=0x7f530813bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f530e01c640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f530809f1c0 0x7f530813bc60 secure :-1 s=READY pgs=5642 cs=0 l=1 rev1=1 crypto rx=0x7f530400f770 tx=0x7f5304003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 10 client.34125.objecter ms_handle_connect 0x7f530809f1c0 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 10 client.34125.objecter resend_mon_ops 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f530400ddc0 con 0x7f530809f1c0 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 10 monclient: dump: 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.270 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f530400ddc0 con 0x7f530809f1c0 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f530809f1c0 msgr2=0x7f530813bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f530809f1c0 0x7f530813bc60 secure :-1 s=READY pgs=5642 cs=0 l=1 rev1=1 crypto rx=0x7f530400f770 tx=0x7f5304003770 comp rx=0 tx=0).stop 2024-09-10T20:30:47.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f530809f1c0 msgr2=0x7f530813bc60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:47.272 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f52feffd640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f530809f1c0 0x7f530813bc60 secure :-1 s=CLOSED pgs=5642 cs=0 l=1 rev1=1 crypto rx=0x7f530400f770 tx=0x7f5304003770 comp rx=0 tx=0).stop 2024-09-10T20:30:47.272 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:30:47.272 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:30:47.272 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:30:47.272 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:30:47.272 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 26, 2024-09-10T20:30:47.272 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 293898, 2024-09-10T20:30:47.272 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:30:47.272 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:47.273 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:30:47.274 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:47.275 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:30:47.276 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:47.277 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:30:47.278 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.279 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:47.280 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.282 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.283 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:30:47.284 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-10T20:30:47.285 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 7 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:47.286 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:47.287 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:47.287 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:47.287 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:47.287 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:47.287 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:47.287 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:30:47.287 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 10 librados: watch_flush enter 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 10 librados: watch_flush exit 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 20 client.34125.objecter shutdown clearing up homeless session... 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 10 client.34125.objecter successfully canceled tick 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 >> v2:172.21.15.176:6812/3003473398 conn(0x7f52ec03cea0 msgr2=0x7f52ec03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 1 --2- 172.21.15.153:0/2470513821 >> v2:172.21.15.176:6812/3003473398 conn(0x7f52ec03cea0 0x7f52ec03f360 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7f530806a960 tx=0x7f52f80a6000 comp rx=0 tx=0).stop 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 10 monclient: shutdown 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f530813e1b0 msgr2=0x7f530809e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 1 --2- 172.21.15.153:0/2470513821 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f530813e1b0 0x7f530809e740 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f530000caf0 tx=0x7f530000a060 comp rx=0 tx=0).stop 2024-09-10T20:30:47.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f530e81d640 1 -- 172.21.15.153:0/2470513821 reap_dead start 2024-09-10T20:30:47.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f530e81d640 1 -- 172.21.15.153:0/2470513821 reap_dead start 2024-09-10T20:30:47.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 shutdown_connections 2024-09-10T20:30:47.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 >> 172.21.15.153:0/2470513821 conn(0x7f53080aba00 msgr2=0x7f5308141120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:47.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 shutdown_connections 2024-09-10T20:30:47.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 1 -- 172.21.15.153:0/2470513821 wait complete. 2024-09-10T20:30:47.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.256+0000 7f5310280640 1 librados: shutdown 2024-09-10T20:30:47.290 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:30:47.290 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T20:30:47.291 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T20:30:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.396+0000 7f45b5632640 10 monclient: get_monmap_and_config 2024-09-10T20:30:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.396+0000 7f45b5632640 10 monclient: build_initial_monmap 2024-09-10T20:30:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.396+0000 7f45b5632640 10 monclient: monmap: 2024-09-10T20:30:47.403 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:47.401086+0000 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:47.401086+0000 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:47.404 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 1 Processor -- start 2024-09-10T20:30:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 1 -- start start 2024-09-10T20:30:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient: init 2024-09-10T20:30:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient: _add_conns ranks=[2,0,3,4,1] 2024-09-10T20:30:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b01368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient(hunting): picked mon.noname-b con 0x7f45b01364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f45b01360a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:30:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45b00a1990 0x7f45b009d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient(hunting): picked mon.noname-a con 0x7f45b00a1990 addr v2:172.21.15.153:3300/0 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45b00a27a0 con 0x7f45b01360a0 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f45b00a2a80 con 0x7f45b00a1990 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45b00a2d60 con 0x7f45b01364f0 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45b5632640 10 monclient(hunting): authenticate will time out at 1207.541013s 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45af7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45b00a1990 0x7f45b009d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b01368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45a67fc640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f45b01360a0 0x7f45b009ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44446/0 (socket says 172.21.15.153:44446) 2024-09-10T20:30:47.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45a67fc640 1 -- 172.21.15.153:0/579544161 learned_addr learned my addr 172.21.15.153:0/579544161 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45af7fe640 10 monclient(hunting): get_auth_request con 0x7f45b00a1990 auth_method 0 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient(hunting): get_auth_request con 0x7f45b01364f0 auth_method 0 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.409 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient(hunting): handle_auth_done global_id 24476 payload 274 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 1 -- 172.21.15.153:0/579544161 >> v2:172.21.15.153:3300/0 conn(0x7f45b00a1990 msgr2=0x7f45b009d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 1 --2- 172.21.15.153:0/579544161 >> v2:172.21.15.153:3300/0 conn(0x7f45b00a1990 0x7f45b009d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 1 -- 172.21.15.153:0/579544161 >> v1:172.21.15.153:6789/0 conn(0x7f45b01360a0 legacy=0x7f45b009ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient: _finish_hunting 0 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 1 monclient: found mon.noname-b 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45aeffd640 1 -- 172.21.15.153:0/579544161 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b0136e10 con 0x7f45b01364f0 2024-09-10T20:30:47.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.400+0000 7f45af7fe640 1 --2- 172.21.15.153:0/579544161 >> v2:172.21.15.153:3300/0 conn(0x7f45b00a1990 0x7f45b009d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45aeffd640 1 -- 172.21.15.153:0/579544161 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 msgr2=0x7f45b01368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45aeffd640 1 --2- 172.21.15.153:0/579544161 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b01368d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f45a00097c0 tx=0x7f45a001c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ccdd20c7dfb53471 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 1 -- 172.21.15.153:0/579544161 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a00b0070 con 0x7f45b01364f0 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 10 monclient: dump: 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.414 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 10 monclient: _finish_auth 0 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 5 monclient: authenticate success, global_id 24476 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 1 -- 172.21.15.153:0/579544161 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45b009dc70 con 0x7f45b01364f0 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:47.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 4 set_mon_vals no callback set 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 1 -- 172.21.15.153:0/579544161 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45a00ab820 con 0x7f45b01364f0 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45ae7fc640 10 monclient: dump: 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.416 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.417 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 10 monclient: shutdown 2024-09-10T20:30:47.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:47.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 1 -- 172.21.15.153:0/579544161 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 msgr2=0x7f45b01368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 1 --2- 172.21.15.153:0/579544161 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b01368d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f45a00097c0 tx=0x7f45a001c360 comp rx=0 tx=0).stop 2024-09-10T20:30:47.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 1 -- 172.21.15.153:0/579544161 shutdown_connections 2024-09-10T20:30:47.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 1 --2- 172.21.15.153:0/579544161 >> v2:172.21.15.153:3300/0 conn(0x7f45b00a1990 0x7f45b009d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 1 --2- 172.21.15.153:0/579544161 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b01368d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 1 -- 172.21.15.153:0/579544161 >> 172.21.15.153:0/579544161 conn(0x7f45b00a71b0 msgr2=0x7f45b00a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 1 -- 172.21.15.153:0/579544161 shutdown_connections 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.404+0000 7f45b5632640 1 -- 172.21.15.153:0/579544161 wait complete. 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient: build_initial_monmap 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient: monmap: 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:47.418 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:47.419 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:47.419 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 librados: starting msgr at 2024-09-10T20:30:47.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 librados: starting objecter 2024-09-10T20:30:47.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 Processor -- start 2024-09-10T20:30:47.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 -- start start 2024-09-10T20:30:47.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 librados: setting wanted keys 2024-09-10T20:30:47.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 librados: calling monclient init 2024-09-10T20:30:47.419 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient: init 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00a1990 0x7f45b0070b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient(hunting): picked mon.noname-b con 0x7f45b00a1990 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45b01360a0 0x7f45b00710c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient(hunting): picked mon.noname-a con 0x7f45b01360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b0071600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient(hunting): picked mon.noname-c con 0x7f45b01364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45b0140ff0 con 0x7f45b01360a0 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f45b0140e70 con 0x7f45b00a1990 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:47.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45b0141170 con 0x7f45b01364f0 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45b5632640 10 monclient(hunting): authenticate will time out at 1207.549050s 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45b01360a0 0x7f45b00710c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45af7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b0071600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45b01360a0 0x7f45b00710c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49742/0 (socket says 172.21.15.153:49742) 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 -- 172.21.15.153:0/401572295 learned_addr learned my addr 172.21.15.153:0/401572295 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient(hunting): get_auth_request con 0x7f45b01360a0 auth_method 0 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45af7fe640 10 monclient(hunting): get_auth_request con 0x7f45b01364f0 auth_method 0 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45af7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:47.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45af7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45af7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45af7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45af7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45af7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45aeffd640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00a1990 0x7f45b0070b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient(hunting): handle_auth_done global_id 34131 payload 824 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 -- 172.21.15.153:0/401572295 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 msgr2=0x7f45b0071600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b0071600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 -- 172.21.15.153:0/401572295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00a1990 msgr2=0x7f45b0070b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00a1990 0x7f45b0070b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient: _finish_hunting 0 2024-09-10T20:30:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 monclient: found mon.noname-a 2024-09-10T20:30:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 -- 172.21.15.153:0/401572295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45b013dd50 con 0x7f45b01360a0 2024-09-10T20:30:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45aeffd640 10 monclient: get_auth_request con 0x7f45b00a1990 auth_method 0 2024-09-10T20:30:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45af7fe640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b0071600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:30:47.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45aeffd640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00a1990 0x7f45b0070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:30:47.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 -- 172.21.15.153:0/401572295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45b01360a0 msgr2=0x7f45b00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:47.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a67fc640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45b01360a0 0x7f45b00710c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f459800c470 tx=0x7f459800c870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a7fff640 10 client.?.objecter ms_handle_connect 0x7f45b01360a0 2024-09-10T20:30:47.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a7fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:47.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a7fff640 1 -- 172.21.15.153:0/401572295 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598017070 con 0x7f45b01360a0 2024-09-10T20:30:47.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:30:47.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a7fff640 10 monclient: dump: 2024-09-10T20:30:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.427 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a7fff640 10 monclient: _finish_auth 0 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a7fff640 10 monclient: _check_auth_tickets 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.408+0000 7f45a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 1 -- 172.21.15.153:0/401572295 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45980070d0 con 0x7f45b01360a0 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 5 monclient: authenticate success, global_id 34131 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 10 monclient: _renew_subs 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45b0074490 con 0x7f45b01360a0 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 1 -- 172.21.15.153:0/401572295 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4598007500 con 0x7f45b01360a0 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:47.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 10 monclient: dump: 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45adffb640 4 set_mon_vals no callback set 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 10 client.34131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 10 monclient: _renew_subs 2024-09-10T20:30:47.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45b013df50 con 0x7f45b01360a0 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 1 librados: init done 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45b01368d0 con 0x7f45b01360a0 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 1 -- 172.21.15.153:0/401572295 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f459801a070 con 0x7f45b01360a0 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 1 --2- 172.21.15.153:0/401572295 >> v2:172.21.15.176:6812/3003473398 conn(0x7f459003cea0 0x7f459003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 1 -- 172.21.15.153:0/401572295 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(204..204 src has 1..204) ==== 4901+0+0 (secure 0 0 0) 0x7f459804ed10 con 0x7f45b01360a0 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 10 client.34131.objecter ms_dispatch 0x7f45b0000ba0 osd_map(204..204 src has 1..204) 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 3 client.34131.objecter handle_osd_map got epochs [204,204] > 0 2024-09-10T20:30:47.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 3 client.34131.objecter handle_osd_map decoding full epoch 204 2024-09-10T20:30:47.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45a7fff640 20 client.34131.objecter dump_active .. 0 homeless 2024-09-10T20:30:47.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45aeffd640 1 --2- 172.21.15.153:0/401572295 >> v2:172.21.15.176:6812/3003473398 conn(0x7f459003cea0 0x7f459003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:47.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.412+0000 7f45aeffd640 10 monclient: get_auth_request con 0x7f459003cea0 auth_method 0 2024-09-10T20:30:47.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.416+0000 7f45aeffd640 1 --2- 172.21.15.153:0/401572295 >> v2:172.21.15.176:6812/3003473398 conn(0x7f459003cea0 0x7f459003f360 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f45a00ab000 tx=0x7f45a0008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:47.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.416+0000 7f45a7fff640 10 client.34131.objecter ms_handle_connect 0x7f459003cea0 2024-09-10T20:30:47.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.416+0000 7f45a7fff640 1 -- 172.21.15.153:0/401572295 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f4598055080 con 0x7f45b01360a0 2024-09-10T20:30:47.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.416+0000 7f45a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:47.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.416+0000 7f45a7fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.640+0000 7f45b5632640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T20:30:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.640+0000 7f45b5632640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T20:30:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.640+0000 7f45b5632640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:47.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.640+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f45b0072f70 con 0x7f45b01360a0 2024-09-10T20:30:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.640+0000 7f45a7fff640 1 -- 172.21.15.153:0/401572295 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f45b013dd50 con 0x7f45b01360a0 2024-09-10T20:30:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T20:30:47.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45a7fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:47.648 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=283817724,config=3541143443,health=1133115229,kv=656860464,logm=277048593}) 2024-09-10T20:30:47.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=10,config=2,health=10,kv=10,logm=68} crc {auth=647142272,config=3541143443,health=2637782757,kv=3615289851,logm=3649728850}) 2024-09-10T20:30:47.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=9,config=2,health=12,kv=10,logm=67} crc {auth=283817724,config=3541143443,health=1133115229,kv=656860464,logm=277048593}) 2024-09-10T20:30:47.649 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=1,health=11,kv=9,logm=70} crc {auth=3451140270,config=555582061,health=3045050150,kv=3749893338,logm=303805302}) 2024-09-10T20:30:47.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 10 librados: watch_flush enter 2024-09-10T20:30:47.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 10 librados: watch_flush exit 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 20 client.34131.objecter shutdown clearing up homeless session... 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 10 client.34131.objecter successfully canceled tick 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 >> v2:172.21.15.176:6812/3003473398 conn(0x7f459003cea0 msgr2=0x7f459003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 --2- 172.21.15.153:0/401572295 >> v2:172.21.15.176:6812/3003473398 conn(0x7f459003cea0 0x7f459003f360 secure :-1 s=READY pgs=3319 cs=0 l=1 rev1=1 crypto rx=0x7f45a00ab000 tx=0x7f45a0008040 comp rx=0 tx=0).stop 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 10 monclient: shutdown 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45b01360a0 msgr2=0x7f45b00710c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45b01360a0 0x7f45b00710c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f459800c470 tx=0x7f459800c870 comp rx=0 tx=0).stop 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 shutdown_connections 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 --2- 172.21.15.153:0/401572295 >> v2:172.21.15.176:6812/3003473398 conn(0x7f459003cea0 0x7f459003f360 unknown :-1 s=CLOSED pgs=3319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45b01364f0 0x7f45b0071600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45b01360a0 0x7f45b00710c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 --2- 172.21.15.153:0/401572295 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45b00a1990 0x7f45b0070b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 >> 172.21.15.153:0/401572295 conn(0x7f45b00a71b0 msgr2=0x7f45b0132a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 shutdown_connections 2024-09-10T20:30:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 -- 172.21.15.153:0/401572295 wait complete. 2024-09-10T20:30:47.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:47.644+0000 7f45b5632640 1 librados: shutdown 2024-09-10T20:30:47.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280178541}) 2024-09-10T20:30:47.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4087683329}) 2024-09-10T20:30:47.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1280178541}) 2024-09-10T20:30:47.653 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.644+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=551018199}) 2024-09-10T20:30:47.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.648+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.648+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425802867}) 2024-09-10T20:30:47.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.648+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2192208763}) 2024-09-10T20:30:47.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.648+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.648+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1425802867}) 2024-09-10T20:30:47.654 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.648+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2969886341}) 2024-09-10T20:30:47.659 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.652+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.659 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.652+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638210360}) 2024-09-10T20:30:47.659 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.652+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3728590742}) 2024-09-10T20:30:47.659 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.652+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.659 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.652+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3638210360}) 2024-09-10T20:30:47.659 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.652+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=780219553}) 2024-09-10T20:30:47.662 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.656+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.662 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.656+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3190651872}) 2024-09-10T20:30:47.662 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.656+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3194997689}) 2024-09-10T20:30:47.662 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.656+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.662 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.656+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3190651872}) 2024-09-10T20:30:47.662 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.656+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=324043064}) 2024-09-10T20:30:47.664 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.665 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241161070}) 2024-09-10T20:30:47.665 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1205259422}) 2024-09-10T20:30:47.665 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.665 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241161070}) 2024-09-10T20:30:47.665 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=469816509}) 2024-09-10T20:30:47.667 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.667 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648133075}) 2024-09-10T20:30:47.667 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3054612979}) 2024-09-10T20:30:47.667 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.667 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1648133075}) 2024-09-10T20:30:47.667 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.660+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=411428288}) 2024-09-10T20:30:47.668 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T20:30:47.669 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.669 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355369266}) 2024-09-10T20:30:47.669 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1899072644}) 2024-09-10T20:30:47.669 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.669 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1355369266}) 2024-09-10T20:30:47.669 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1213542603}) 2024-09-10T20:30:47.671 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.671 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675694431}) 2024-09-10T20:30:47.671 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2600422659}) 2024-09-10T20:30:47.671 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.671 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675694431}) 2024-09-10T20:30:47.671 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.664+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1440292164}) 2024-09-10T20:30:47.672 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.672 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552629568}) 2024-09-10T20:30:47.672 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3635216899}) 2024-09-10T20:30:47.672 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.673 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2552629568}) 2024-09-10T20:30:47.673 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4225424192}) 2024-09-10T20:30:47.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296225190}) 2024-09-10T20:30:47.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2717041730}) 2024-09-10T20:30:47.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296225190}) 2024-09-10T20:30:47.674 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.668+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2178344232}) 2024-09-10T20:30:47.676 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.676 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297719195}) 2024-09-10T20:30:47.676 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2747512129}) 2024-09-10T20:30:47.676 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.676 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=297719195}) 2024-09-10T20:30:47.676 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948035276}) 2024-09-10T20:30:47.677 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.677 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526565791}) 2024-09-10T20:30:47.677 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=28391682}) 2024-09-10T20:30:47.677 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.678 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3526565791}) 2024-09-10T20:30:47.678 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3285187052}) 2024-09-10T20:30:47.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731233816}) 2024-09-10T20:30:47.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=665798527}) 2024-09-10T20:30:47.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3731233816}) 2024-09-10T20:30:47.679 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.672+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2247548676}) 2024-09-10T20:30:47.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.676+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.676+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877676275}) 2024-09-10T20:30:47.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.676+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4031431797}) 2024-09-10T20:30:47.681 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.676+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.676+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1877676275}) 2024-09-10T20:30:47.682 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.676+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=867062907}) 2024-09-10T20:30:47.684 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.684 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629462372}) 2024-09-10T20:30:47.684 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=667335929}) 2024-09-10T20:30:47.684 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.684 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1629462372}) 2024-09-10T20:30:47.684 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2883100214}) 2024-09-10T20:30:47.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578328078}) 2024-09-10T20:30:47.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3559029662}) 2024-09-10T20:30:47.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3578328078}) 2024-09-10T20:30:47.686 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2290659103}) 2024-09-10T20:30:47.687 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.687 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3860216789}) 2024-09-10T20:30:47.687 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=681539053}) 2024-09-10T20:30:47.687 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.688 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3860216789}) 2024-09-10T20:30:47.688 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.680+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=841404680}) 2024-09-10T20:30:47.689 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.689 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928914837}) 2024-09-10T20:30:47.689 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3110305882}) 2024-09-10T20:30:47.689 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.689 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2928914837}) 2024-09-10T20:30:47.689 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1402643976}) 2024-09-10T20:30:47.691 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.691 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844850234}) 2024-09-10T20:30:47.691 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1818490051}) 2024-09-10T20:30:47.691 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.691 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=844850234}) 2024-09-10T20:30:47.691 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.684+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=463620950}) 2024-09-10T20:30:47.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=2,osd_metadata=4,osd_pg_creating=1,osd_snap=9} crc {logm=247474914,mdsmap=3035479434,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=4035573183,mon_config_key=2906107253,monmap=999369809,nvmeofgw=1596038039,osd_metadata=220300310,osd_pg_creating=1871580294,osd_snap=3108115498}) 2024-09-10T20:30:47.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=13,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=4} crc {logm=3840013523,mdsmap=3397941211,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=1367945297,mon_config_key=2002537016,monmap=999369809,nvmeofgw=598337686,osd_metadata=753924036,osd_pg_creating=1871580294,osd_snap=3870860314}) 2024-09-10T20:30:47.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=44,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=13,monmap=3,nvmeofgw=2,osd_metadata=4,osd_pg_creating=1,osd_snap=9} crc {logm=247474914,mdsmap=3035479434,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=4035573183,mon_config_key=2906107253,monmap=999369809,nvmeofgw=1596038039,osd_metadata=220300310,osd_pg_creating=1871580294,osd_snap=3108115498}) 2024-09-10T20:30:47.693 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=43,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=9} crc {logm=1478642730,mdsmap=3397941211,mgr=2368631417,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=2081858523,mon_config_key=3257012160,monmap=999369809,nvmeofgw=598337686,osd_metadata=3795740981,osd_pg_creating=1871580294,osd_snap=3873389711}) 2024-09-10T20:30:47.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=107434784,osdmap=3653082196}) 2024-09-10T20:30:47.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=107434784,osdmap=2469999064}) 2024-09-10T20:30:47.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=107434784,osdmap=3653082196}) 2024-09-10T20:30:47.695 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.688+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=107434784,osdmap=1908735004}) 2024-09-10T20:30:47.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.692+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.692+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1237777595}) 2024-09-10T20:30:47.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.692+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3818674541}) 2024-09-10T20:30:47.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.692+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.692+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1237777595}) 2024-09-10T20:30:47.697 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.692+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1486616519}) 2024-09-10T20:30:47.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.696+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.696+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1678673200}) 2024-09-10T20:30:47.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.696+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3322373688}) 2024-09-10T20:30:47.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.696+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.696+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1678673200}) 2024-09-10T20:30:47.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.696+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1042694147}) 2024-09-10T20:30:47.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.708+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.708+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=1343125319}) 2024-09-10T20:30:47.715 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.708+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=17} crc {osdmap=4072245729}) 2024-09-10T20:30:47.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.708+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:30:47.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.708+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=16} crc {osdmap=1343125319}) 2024-09-10T20:30:47.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:30:47.708+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=13} crc {osdmap=3716585696}) 2024-09-10T20:30:48.668 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T20:30:48.669 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:30:48.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient: get_monmap_and_config 2024-09-10T20:30:48.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient: build_initial_monmap 2024-09-10T20:30:48.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient: monmap: 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:48.791714+0000 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:48.791714+0000 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:48.794 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:48.795 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:48.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 1 Processor -- start 2024-09-10T20:30:48.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 1 -- start start 2024-09-10T20:30:48.796 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient: init 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7cbc0a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient(hunting): picked mon.noname-b con 0x7f7cbc0a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f7cbc0a49a0 0x7f7cbc13fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient(hunting): picked mon.noname-c con 0x7f7cbc0a49a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cbc0a6a20 con 0x7f7cbc0a41a0 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f7cbc0a2570 con 0x7f7cbc0a49a0 2024-09-10T20:30:48.797 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7cbc070400 con 0x7f7cbc0a4580 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc41b6640 10 monclient(hunting): authenticate will time out at 1208.931252s 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44902/0 (socket says 172.21.15.153:44902) 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 1 -- 172.21.15.153:0/714904480 learned_addr learned my addr 172.21.15.153:0/714904480 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient(hunting): get_auth_request con 0x7f7cbc0a4580 auth_method 0 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:48.798 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient(hunting): handle_auth_done global_id 24485 payload 274 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc2753640 1 --2- 172.21.15.153:0/714904480 >> v2:172.21.15.176:3300/0 conn(0x7f7cbc0a49a0 0x7f7cbc13fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 1 -- 172.21.15.153:0/714904480 >> v2:172.21.15.176:3300/0 conn(0x7f7cbc0a49a0 msgr2=0x7f7cbc13fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 1 --2- 172.21.15.153:0/714904480 >> v2:172.21.15.176:3300/0 conn(0x7f7cbc0a49a0 0x7f7cbc13fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 1 -- 172.21.15.153:0/714904480 >> v1:172.21.15.176:6789/0 conn(0x7f7cbc0a41a0 legacy=0x7f7cbc0a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient: _finish_hunting 0 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 1 monclient: found mon.noname-b 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc1751640 1 -- 172.21.15.153:0/714904480 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cbc0a85a0 con 0x7f7cbc0a4580 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc2753640 10 monclient: get_auth_request con 0x7f7cbc0a49a0 auth_method 0 2024-09-10T20:30:48.799 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.788+0000 7f7cc2753640 1 --2- 172.21.15.153:0/714904480 >> v2:172.21.15.176:3300/0 conn(0x7f7cbc0a49a0 0x7f7cbc13fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:30:48.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.792+0000 7f7cc1751640 1 -- 172.21.15.153:0/714904480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 msgr2=0x7f7cbc13d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:48.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.792+0000 7f7cc1751640 1 --2- 172.21.15.153:0/714904480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13d7c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009e50 tx=0x7f7cb801c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fce76255824c036 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 1 -- 172.21.15.153:0/714904480 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80a5730 con 0x7f7cbc0a4580 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 10 monclient: dump: 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:48.804 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 10 monclient: _finish_auth 0 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 10 monclient: _check_auth_tickets 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 1 -- 172.21.15.153:0/714904480 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb80a58d0 con 0x7f7cbc0a4580 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 1 -- 172.21.15.153:0/714904480 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80a5ca0 con 0x7f7cbc0a4580 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 5 monclient: authenticate success, global_id 24485 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 4 set_mon_vals no callback set 2024-09-10T20:30:48.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc0f50640 10 monclient: dump: 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:48.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient: shutdown 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 -- 172.21.15.153:0/714904480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 msgr2=0x7f7cbc13d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 --2- 172.21.15.153:0/714904480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13d7c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7cb8009e50 tx=0x7f7cb801c8f0 comp rx=0 tx=0).stop 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 -- 172.21.15.153:0/714904480 shutdown_connections 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 --2- 172.21.15.153:0/714904480 >> v2:172.21.15.176:3300/0 conn(0x7f7cbc0a49a0 0x7f7cbc13fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 --2- 172.21.15.153:0/714904480 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13d7c0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 -- 172.21.15.153:0/714904480 >> 172.21.15.153:0/714904480 conn(0x7f7cbc0ab7e0 msgr2=0x7f7cbc0a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 -- 172.21.15.153:0/714904480 shutdown_connections 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 -- 172.21.15.153:0/714904480 wait complete. 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient: build_initial_monmap 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient: monmap: 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:48.807 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:48.808 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:48.808 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:48.808 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:48.808 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:48.808 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:48.808 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:48.808 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:48.808 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:48.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 librados: starting msgr at 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 librados: starting objecter 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 Processor -- start 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 -- start start 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 librados: setting wanted keys 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 librados: calling monclient init 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient: init 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cbc0a41a0 0x7f7cbc1284c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient(hunting): picked mon.noname-a con 0x7f7cbc0a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient(hunting): picked mon.noname-c con 0x7f7cbc0a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:48.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cbc0a49a0 0x7f7cbc1d01c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient(hunting): picked mon.noname-b con 0x7f7cbc0a49a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc142e00 con 0x7f7cbc0a41a0 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7cbc142c80 con 0x7f7cbc0a49a0 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7cbc142f80 con 0x7f7cbc0a4580 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc41b6640 10 monclient(hunting): authenticate will time out at 1208.938563s 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44910/0 (socket says 172.21.15.153:44910) 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 1 -- 172.21.15.153:0/415468006 learned_addr learned my addr 172.21.15.153:0/415468006 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:48.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient(hunting): get_auth_request con 0x7f7cbc0a4580 auth_method 0 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1f52640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cbc0a41a0 0x7f7cbc1284c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1f52640 10 monclient(hunting): get_auth_request con 0x7f7cbc0a41a0 auth_method 0 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1f52640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1f52640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1f52640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1f52640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1f52640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:48.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1f52640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc2753640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cbc0a49a0 0x7f7cbc1d01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc2753640 10 monclient(hunting): get_auth_request con 0x7f7cbc0a49a0 auth_method 0 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc2753640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc2753640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc2753640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient(hunting): handle_auth_done global_id 24488 payload 824 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 1 -- 172.21.15.153:0/415468006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cbc0a49a0 msgr2=0x7f7cbc1d01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cbc0a49a0 0x7f7cbc1d01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 1 -- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cbc0a41a0 msgr2=0x7f7cbc1284c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cbc0a41a0 0x7f7cbc1284c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient: _finish_hunting 0 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 1 monclient: found mon.noname-c 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:48.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:30:48.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1751640 1 -- 172.21.15.153:0/415468006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cbc1299e0 con 0x7f7cbc0a4580 2024-09-10T20:30:48.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.796+0000 7f7cc1f52640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cbc0a41a0 0x7f7cbc1284c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:30:48.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc1751640 1 -- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 msgr2=0x7f7cbc13f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:48.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc1751640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13f330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7cb801c720 tx=0x7f7cb8019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:48.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 client.?.objecter ms_handle_connect 0x7f7cbc0a4580 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 1 -- 172.21.15.153:0/415468006 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80bc020 con 0x7f7cbc0a4580 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 monclient: dump: 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:48.816 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 monclient: _finish_auth 0 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 1 -- 172.21.15.153:0/415468006 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cb80b8a90 con 0x7f7cbc0a4580 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 1 -- 172.21.15.153:0/415468006 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cb80b8e60 con 0x7f7cbc0a4580 2024-09-10T20:30:48.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 5 monclient: authenticate success, global_id 24488 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 monclient: dump: 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:48.818 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 10 monclient: _renew_subs 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7caf7fe640 4 set_mon_vals no callback set 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cbc0a21c0 con 0x7f7cbc0a4580 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 10 client.24488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 10 monclient: _renew_subs 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cbc127960 con 0x7f7cbc0a4580 2024-09-10T20:30:48.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 1 librados: init done 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cbc0a3740 con 0x7f7cbc0a4580 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 1 -- 172.21.15.153:0/415468006 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f7cb80bf020 con 0x7f7cbc0a4580 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 1 --2- 172.21.15.153:0/415468006 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ca003ced0 0x7f7ca003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 1 -- 172.21.15.153:0/415468006 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(205..205 src has 1..205) ==== 5204+0+0 (secure 0 0 0) 0x7f7cb80eaf50 con 0x7f7cbc0a4580 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 10 client.24488.objecter ms_dispatch 0x7f7cbc000ba0 osd_map(205..205 src has 1..205) 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 3 client.24488.objecter handle_osd_map got epochs [205,205] > 0 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 3 client.24488.objecter handle_osd_map decoding full epoch 205 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cae7fc640 20 client.24488.objecter dump_active .. 0 homeless 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc1f52640 1 --2- 172.21.15.153:0/415468006 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ca003ced0 0x7f7ca003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:48.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.800+0000 7f7cc1f52640 10 monclient: get_auth_request con 0x7f7ca003ced0 auth_method 0 2024-09-10T20:30:48.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.804+0000 7f7cae7fc640 10 client.24488.objecter ms_handle_connect 0x7f7ca003ced0 2024-09-10T20:30:48.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.804+0000 7f7cc1f52640 1 --2- 172.21.15.153:0/415468006 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ca003ced0 0x7f7ca003f390 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0007fd0 tx=0x7f7cb0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:48.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.808+0000 7f7cae7fc640 1 -- 172.21.15.153:0/415468006 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f7cb80c6070 con 0x7f7cbc0a4580 2024-09-10T20:30:48.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.808+0000 7f7cae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:48.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:48.808+0000 7f7cae7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:49.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.048+0000 7f7cc41b6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:30:49.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.048+0000 7f7cc41b6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:30:49.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.048+0000 7f7cc41b6640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:49.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.048+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7cbc06cd40 con 0x7f7cbc0a4580 2024-09-10T20:30:49.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cae7fc640 1 -- 172.21.15.153:0/415468006 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f7cb8016600 con 0x7f7cbc0a4580 2024-09-10T20:30:49.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:30:49.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cae7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:49.056 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":26,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 10 librados: watch_flush enter 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 10 librados: watch_flush exit 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 20 client.24488.objecter shutdown clearing up homeless session... 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 10 client.24488.objecter successfully canceled tick 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ca003ced0 msgr2=0x7f7ca003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 --2- 172.21.15.153:0/415468006 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ca003ced0 0x7f7ca003f390 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7f7cb0007fd0 tx=0x7f7cb0009040 comp rx=0 tx=0).stop 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 10 monclient: shutdown 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 msgr2=0x7f7cbc13f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13f330 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7cb801c720 tx=0x7f7cb8019690 comp rx=0 tx=0).stop 2024-09-10T20:30:49.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 shutdown_connections 2024-09-10T20:30:49.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7cbc0a49a0 0x7f7cbc1d01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 --2- 172.21.15.153:0/415468006 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7ca003ced0 0x7f7ca003f390 unknown :-1 s=CLOSED pgs=3321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7cbc0a4580 0x7f7cbc13f330 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 --2- 172.21.15.153:0/415468006 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7cbc0a41a0 0x7f7cbc1284c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 >> 172.21.15.153:0/415468006 conn(0x7f7cbc0ab7e0 msgr2=0x7f7cbc0a0470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:49.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 shutdown_connections 2024-09-10T20:30:49.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 -- 172.21.15.153:0/415468006 wait complete. 2024-09-10T20:30:49.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.052+0000 7f7cc41b6640 1 librados: shutdown 2024-09-10T20:30:49.074 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T20:30:49.074 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T20:30:49.074 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T20:30:49.074 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:30:49.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient: get_monmap_and_config 2024-09-10T20:30:49.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient: build_initial_monmap 2024-09-10T20:30:49.192 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient: monmap: 2024-09-10T20:30:49.192 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:49.193 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.193 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:49.190427+0000 2024-09-10T20:30:49.193 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:49.190427+0000 2024-09-10T20:30:49.193 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:49.193 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.193 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:49.193 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:49.193 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:49.193 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:49.194 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:49.194 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 1 Processor -- start 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 1 -- start start 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient: init 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient: _add_conns ranks=[0,4,1,2,3] 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d9c1364f0 addr v1:172.21.15.153:6789/0 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f2d9c1360a0 0x7f2d9c13b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient(hunting): picked mon.noname-c con 0x7f2d9c1360a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d9c0a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d9c0a27a0 con 0x7f2d9c1364f0 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d9c0a2a80 con 0x7f2d9c0a1990 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.196 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f2d9c0a2550 con 0x7f2d9c1360a0 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2da1bed640 10 monclient(hunting): authenticate will time out at 1209.330088s 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9b7fe640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2d9c1364f0 0x7f2d9c1368d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44460/0 (socket says 172.21.15.153:44460) 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9b7fe640 1 -- 172.21.15.153:0/1035929613 learned_addr learned my addr 172.21.15.153:0/1035929613 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 1 --2- 172.21.15.153:0/1035929613 >> v2:172.21.15.176:3300/0 conn(0x7f2d9c1360a0 0x7f2d9c13b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient(hunting): get_auth_request con 0x7f2d9c1360a0 auth_method 0 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient(hunting): handle_auth_done global_id 20008 payload 274 2024-09-10T20:30:49.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 1 -- 172.21.15.153:0/1035929613 >> v1:172.21.15.176:6789/0 conn(0x7f2d9c0a1990 legacy=0x7f2d9c13d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:49.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 1 -- 172.21.15.153:0/1035929613 >> v1:172.21.15.153:6789/0 conn(0x7f2d9c1364f0 legacy=0x7f2d9c1368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:49.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient: _finish_hunting 0 2024-09-10T20:30:49.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 1 monclient: found mon.noname-c 2024-09-10T20:30:49.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:49.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:30:49.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.188+0000 7f2d9affd640 1 -- 172.21.15.153:0/1035929613 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c0a2d60 con 0x7f2d9c1360a0 2024-09-10T20:30:49.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.192+0000 7f2d9affd640 1 -- 172.21.15.153:0/1035929613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 msgr2=0x7f2d9c13b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.192+0000 7f2d9affd640 1 --2- 172.21.15.153:0/1035929613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 0x7f2d9c13b420 secure :-1 s=READY pgs=5648 cs=0 l=1 rev1=1 crypto rx=0x7f2d900097c0 tx=0x7f2d9001c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56ffa9c5f9c8c512 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 1 -- 172.21.15.153:0/1035929613 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d900b0070 con 0x7f2d9c1360a0 2024-09-10T20:30:49.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:30:49.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 10 monclient: dump: 2024-09-10T20:30:49.200 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.200 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.200 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.200 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.200 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 10 monclient: _finish_auth 0 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 1 -- 172.21.15.153:0/1035929613 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d900ab600 con 0x7f2d9c1360a0 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 1 -- 172.21.15.153:0/1035929613 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d900ab9d0 con 0x7f2d9c1360a0 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:30:49.201 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9a7fc640 10 monclient: dump: 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 5 monclient: authenticate success, global_id 20008 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:49.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 4 set_mon_vals no callback set 2024-09-10T20:30:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient: shutdown 2024-09-10T20:30:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 -- 172.21.15.153:0/1035929613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 msgr2=0x7f2d9c13b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 --2- 172.21.15.153:0/1035929613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 0x7f2d9c13b420 secure :-1 s=READY pgs=5648 cs=0 l=1 rev1=1 crypto rx=0x7f2d900097c0 tx=0x7f2d9001c360 comp rx=0 tx=0).stop 2024-09-10T20:30:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 -- 172.21.15.153:0/1035929613 shutdown_connections 2024-09-10T20:30:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 --2- 172.21.15.153:0/1035929613 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 0x7f2d9c13b420 unknown :-1 s=CLOSED pgs=5648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 -- 172.21.15.153:0/1035929613 >> 172.21.15.153:0/1035929613 conn(0x7f2d9c0a71b0 msgr2=0x7f2d9c0a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 -- 172.21.15.153:0/1035929613 shutdown_connections 2024-09-10T20:30:49.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 -- 172.21.15.153:0/1035929613 wait complete. 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient: build_initial_monmap 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient: monmap: 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 librados: starting msgr at 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 librados: starting objecter 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 Processor -- start 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 -- start start 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 librados: setting wanted keys 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 librados: calling monclient init 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient: init 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9c0a1990 0x7f2d9c1370d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient(hunting): picked mon.noname-c con 0x7f2d9c0a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 0x7f2d9c137610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient(hunting): picked mon.noname-b con 0x7f2d9c1360a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c1364f0 0x7f2d9c137b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient(hunting): picked mon.noname-a con 0x7f2d9c1364f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c09e7b0 con 0x7f2d9c1364f0 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c09e630 con 0x7f2d9c1360a0 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2d9c09e930 con 0x7f2d9c0a1990 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2da1bed640 10 monclient(hunting): authenticate will time out at 1209.338790s 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c1364f0 0x7f2d9c137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9c0a1990 0x7f2d9c1370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c1364f0 0x7f2d9c137b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49772/0 (socket says 172.21.15.153:49772) 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 1 -- 172.21.15.153:0/3744771700 learned_addr learned my addr 172.21.15.153:0/3744771700 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:49.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient(hunting): get_auth_request con 0x7f2d9c1364f0 auth_method 0 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9b7fe640 10 monclient(hunting): get_auth_request con 0x7f2d9c0a1990 auth_method 0 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9affd640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 0x7f2d9c137610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9affd640 10 monclient(hunting): get_auth_request con 0x7f2d9c1360a0 auth_method 0 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9affd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient(hunting): handle_auth_done global_id 34143 payload 824 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9c0a1990 msgr2=0x7f2d9c1370d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9c0a1990 0x7f2d9c1370d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 msgr2=0x7f2d9c137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 0x7f2d9c137610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient: _finish_hunting 0 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 1 monclient: found mon.noname-a 2024-09-10T20:30:49.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9bfff640 1 -- 172.21.15.153:0/3744771700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d9c139990 con 0x7f2d9c1364f0 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9b7fe640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2d9c0a1990 0x7f2d9c1370d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.196+0000 7f2d9affd640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d9c1360a0 0x7f2d9c137610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9bfff640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c1364f0 msgr2=0x7f2d9c137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9bfff640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c1364f0 0x7f2d9c137b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400c780 tx=0x7f2d9400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 client.?.objecter ms_handle_connect 0x7f2d9c1364f0 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9400ac60 con 0x7f2d9c1364f0 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 monclient: dump: 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.215 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 monclient: _finish_auth 0 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 monclient: _check_auth_tickets 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d9400ae00 con 0x7f2d9c1364f0 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:49.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d94004bb0 con 0x7f2d9c1364f0 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 5 monclient: authenticate success, global_id 34143 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 10 monclient: _renew_subs 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d9c133c70 con 0x7f2d9c1364f0 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d99ffb640 4 set_mon_vals no callback set 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 monclient: dump: 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.217 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 10 client.34143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 10 monclient: _renew_subs 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d9c13e880 con 0x7f2d9c1364f0 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 1 librados: init done 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12f520 0x7f2d9c12f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c09e0a0 con 0x7f2d9c12f520 2024-09-10T20:30:49.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12f520 0x7f2d9c12f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 10 monclient: get_auth_request con 0x7f2d9c12f520 auth_method 0 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_done global_id 34146 payload 824 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f2d94005030 con 0x7f2d9c1364f0 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9b7fe640 1 -- 172.21.15.153:0/3744771700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2d9c13e6b0 con 0x7f2d9c12f520 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 1 --2- 172.21.15.153:0/3744771700 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803cef0 0x7f2d8803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(206..206 src has 1..206) ==== 5531+0+0 (secure 0 0 0) 0x7f2d9404fc80 con 0x7f2d9c1364f0 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 10 client.34143.objecter ms_dispatch 0x7f2d9c000ba0 osd_map(206..206 src has 1..206) 2024-09-10T20:30:49.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 3 client.34143.objecter handle_osd_map got epochs [206,206] > 0 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 3 client.34143.objecter handle_osd_map decoding full epoch 206 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d83fff640 20 client.34143.objecter dump_active .. 0 homeless 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9affd640 1 --2- 172.21.15.153:0/3744771700 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803cef0 0x7f2d8803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.200+0000 7f2d9affd640 10 monclient: get_auth_request con 0x7f2d8803cef0 auth_method 0 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d9b7fe640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12f520 msgr2=0x7f2d9c12f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d9b7fe640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12f520 0x7f2d9c12f900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0072a0 tx=0x7f2d8c007760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 10 client.34143.objecter ms_handle_connect 0x7f2d9c12f520 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 10 client.34143.objecter resend_mon_ops 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c00fce0 con 0x7f2d9c12f520 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 10 monclient: dump: 2024-09-10T20:30:49.220 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2d8c00fce0 con 0x7f2d9c12f520 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12f520 msgr2=0x7f2d9c12f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12f520 0x7f2d9c12f900 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0072a0 tx=0x7f2d8c007760 comp rx=0 tx=0).stop 2024-09-10T20:30:49.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12f520 msgr2=0x7f2d9c12f900 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:49.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12f520 0x7f2d9c12f900 secure :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0072a0 tx=0x7f2d8c007760 comp rx=0 tx=0).stop 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d83fff640 10 client.34143.objecter ms_handle_connect 0x7f2d8803cef0 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.204+0000 7f2d9affd640 1 --2- 172.21.15.153:0/3744771700 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803cef0 0x7f2d8803f3b0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f2d900ad000 tx=0x7f2d900a4f90 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2da1bed640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2da1bed640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12fe40 0x7f2d9c06f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2da1bed640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2d9c1368d0 con 0x7f2d9c12fe40 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2d9b7fe640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12fe40 0x7f2d9c06f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2d9b7fe640 10 monclient: get_auth_request con 0x7f2d9c12fe40 auth_method 0 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2d9b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2d9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.208+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d9b7fe640 10 monclient(hunting): handle_auth_done global_id 34149 payload 824 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d9b7fe640 1 -- 172.21.15.153:0/3744771700 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2d9c09fd60 con 0x7f2d9c12fe40 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 10 client.34143.objecter ms_handle_connect 0x7f2d9c12fe40 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 10 client.34143.objecter resend_mon_ops 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d9b7fe640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12fe40 msgr2=0x7f2d9c06f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d9b7fe640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12fe40 0x7f2d9c06f830 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0072a0 tx=0x7f2d8c004950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d8c006030 con 0x7f2d9c12fe40 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 10 monclient: dump: 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f2d8c006030 con 0x7f2d9c12fe40 2024-09-10T20:30:49.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12fe40 msgr2=0x7f2d9c06f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12fe40 0x7f2d9c06f830 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0072a0 tx=0x7f2d8c004950 comp rx=0 tx=0).stop 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12fe40 msgr2=0x7f2d9c06f830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d83fff640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c12fe40 0x7f2d9c06f830 secure :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2d8c0072a0 tx=0x7f2d8c004950 comp rx=0 tx=0).stop 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 26, 2024-09-10T20:30:49.227 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 9771, 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:49.228 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:49.229 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:30:49.230 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:49.231 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:30:49.232 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.234 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:30:49.235 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.237 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:49.238 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:30:49.239 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.239 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.239 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:30:49.239 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:30:49.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 10 librados: watch_flush enter 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 10 librados: watch_flush exit 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 20 client.34143.objecter shutdown clearing up homeless session... 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 10 client.34143.objecter successfully canceled tick 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803cef0 msgr2=0x7f2d8803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 1 --2- 172.21.15.153:0/3744771700 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803cef0 0x7f2d8803f3b0 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f2d900ad000 tx=0x7f2d900a4f90 comp rx=0 tx=0).stop 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 10 monclient: shutdown 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c1364f0 msgr2=0x7f2d9c137b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 1 --2- 172.21.15.153:0/3744771700 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d9c1364f0 0x7f2d9c137b50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2d9400c780 tx=0x7f2d9400cc40 comp rx=0 tx=0).stop 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d9bfff640 1 -- 172.21.15.153:0/3744771700 reap_dead start 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2d9bfff640 1 -- 172.21.15.153:0/3744771700 reap_dead start 2024-09-10T20:30:49.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 shutdown_connections 2024-09-10T20:30:49.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 >> 172.21.15.153:0/3744771700 conn(0x7f2d9c0a71b0 msgr2=0x7f2d9c131e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:49.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 shutdown_connections 2024-09-10T20:30:49.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 1 -- 172.21.15.153:0/3744771700 wait complete. 2024-09-10T20:30:49.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.212+0000 7f2da1bed640 1 librados: shutdown 2024-09-10T20:30:49.242 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:30:49.242 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T20:30:49.242 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T20:30:49.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient: get_monmap_and_config 2024-09-10T20:30:49.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient: build_initial_monmap 2024-09-10T20:30:49.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient: monmap: 2024-09-10T20:30:49.361 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:49.361 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.361 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:49.359157+0000 2024-09-10T20:30:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:49.359157+0000 2024-09-10T20:30:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:49.362 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:49.363 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:49.363 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 1 Processor -- start 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 1 -- start start 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient: init 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f273409d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient(hunting): picked mon.noname-b con 0x7f27340a28c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f27340a3060 addr v1:172.21.15.153:6789/0 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f273413a7c0 0x7f273413cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient(hunting): picked mon.noname-c con 0x7f273413a7c0 addr v2:172.21.15.176:3300/0 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f273409f890 con 0x7f27340a3060 2024-09-10T20:30:49.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f273409f640 con 0x7f273413a7c0 2024-09-10T20:30:49.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f27340a34e0 con 0x7f27340a28c0 2024-09-10T20:30:49.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:49.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273bd8b640 10 monclient(hunting): authenticate will time out at 1209.498885s 2024-09-10T20:30:49.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f273409d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739326640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f27340a3060 0x7f273409d570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44470/0 (socket says 172.21.15.153:44470) 2024-09-10T20:30:49.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739326640 1 -- 172.21.15.153:0/3177082365 learned_addr learned my addr 172.21.15.153:0/3177082365 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:30:49.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient(hunting): get_auth_request con 0x7f27340a28c0 auth_method 0 2024-09-10T20:30:49.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273a328640 1 --2- 172.21.15.153:0/3177082365 >> v2:172.21.15.176:3300/0 conn(0x7f273413a7c0 0x7f273413cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273a328640 10 monclient(hunting): get_auth_request con 0x7f273413a7c0 auth_method 0 2024-09-10T20:30:49.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273a328640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273a328640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f273a328640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient(hunting): handle_auth_done global_id 24494 payload 274 2024-09-10T20:30:49.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 1 -- 172.21.15.153:0/3177082365 >> v2:172.21.15.176:3300/0 conn(0x7f273413a7c0 msgr2=0x7f273413cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:49.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 1 --2- 172.21.15.153:0/3177082365 >> v2:172.21.15.176:3300/0 conn(0x7f273413a7c0 0x7f273413cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 1 -- 172.21.15.153:0/3177082365 >> v1:172.21.15.153:6789/0 conn(0x7f27340a3060 legacy=0x7f273409d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:49.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient: _finish_hunting 0 2024-09-10T20:30:49.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 1 monclient: found mon.noname-b 2024-09-10T20:30:49.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:49.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:30:49.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.356+0000 7f2739b27640 1 -- 172.21.15.153:0/3177082365 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273409e350 con 0x7f27340a28c0 2024-09-10T20:30:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2739b27640 1 -- 172.21.15.153:0/3177082365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 msgr2=0x7f273409d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2739b27640 1 --2- 172.21.15.153:0/3177082365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f273409d030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f27300097c0 tx=0x7f273001c330 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c5d444c6cea48f4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 1 -- 172.21.15.153:0/3177082365 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27300b0070 con 0x7f27340a28c0 2024-09-10T20:30:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:30:49.378 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 10 monclient: dump: 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 10 monclient: _finish_auth 0 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 10 monclient: _check_auth_tickets 2024-09-10T20:30:49.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 1 -- 172.21.15.153:0/3177082365 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27300ab600 con 0x7f27340a28c0 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f273bd8b640 5 monclient: authenticate success, global_id 24494 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f273bd8b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f273bd8b640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f273bd8b640 4 set_mon_vals no callback set 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 1 -- 172.21.15.153:0/3177082365 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27300ab9d0 con 0x7f27340a28c0 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.360+0000 7f2738b25640 10 monclient: dump: 2024-09-10T20:30:49.380 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient: shutdown 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:49.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 -- 172.21.15.153:0/3177082365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 msgr2=0x7f273409d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 --2- 172.21.15.153:0/3177082365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f273409d030 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f27300097c0 tx=0x7f273001c330 comp rx=0 tx=0).stop 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 -- 172.21.15.153:0/3177082365 shutdown_connections 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 --2- 172.21.15.153:0/3177082365 >> v2:172.21.15.176:3300/0 conn(0x7f273413a7c0 0x7f273413cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 --2- 172.21.15.153:0/3177082365 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f273409d030 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 -- 172.21.15.153:0/3177082365 >> 172.21.15.153:0/3177082365 conn(0x7f27340a71b0 msgr2=0x7f27340a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 -- 172.21.15.153:0/3177082365 shutdown_connections 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 -- 172.21.15.153:0/3177082365 wait complete. 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient: build_initial_monmap 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient: monmap: 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:49.382 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:49.383 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:49.383 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.383 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:49.383 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:49.383 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:49.383 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 librados: starting msgr at 2024-09-10T20:30:49.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 librados: starting objecter 2024-09-10T20:30:49.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 Processor -- start 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 -- start start 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 librados: setting wanted keys 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 librados: calling monclient init 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient: init 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f2734072ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient(hunting): picked mon.noname-c con 0x7f27340a28c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27340a3060 0x7f2734073420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient(hunting): picked mon.noname-a con 0x7f27340a3060 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:49.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f273413a7c0 0x7f2734073960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient(hunting): picked mon.noname-b con 0x7f273413a7c0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f27340a1930 con 0x7f27340a3060 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f27340a17b0 con 0x7f273413a7c0 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f27340a1ab0 con 0x7f27340a28c0 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273bd8b640 10 monclient(hunting): authenticate will time out at 1209.505924s 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f2734072ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739326640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27340a3060 0x7f2734073420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f2734072ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:44946/0 (socket says 172.21.15.153:44946) 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 1 -- 172.21.15.153:0/1528800753 learned_addr learned my addr 172.21.15.153:0/1528800753 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient(hunting): get_auth_request con 0x7f27340a28c0 auth_method 0 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739326640 10 monclient(hunting): get_auth_request con 0x7f27340a3060 auth_method 0 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739326640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739326640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739326640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739326640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739326640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739326640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient(hunting): handle_auth_done global_id 24497 payload 824 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f273a328640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f273413a7c0 0x7f2734073960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f273413a7c0 msgr2=0x7f2734073960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f273413a7c0 0x7f2734073960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27340a3060 msgr2=0x7f2734073420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27340a3060 0x7f2734073420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient: _finish_hunting 0 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 1 monclient: found mon.noname-c 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739b27640 1 -- 172.21.15.153:0/1528800753 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273411f810 con 0x7f27340a28c0 2024-09-10T20:30:49.387 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.364+0000 7f2739326640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f27340a3060 0x7f2734073420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739b27640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 msgr2=0x7f2734072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739b27640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f2734072ee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f273001cbd0 tx=0x7f27300190d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 client.?.objecter ms_handle_connect 0x7f27340a28c0 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27300b8020 con 0x7f27340a28c0 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 monclient: dump: 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.388 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 monclient: _finish_auth 0 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27300b4840 con 0x7f27340a28c0 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27300b0040 con 0x7f27340a28c0 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 5 monclient: authenticate success, global_id 24497 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 10 monclient: _renew_subs 2024-09-10T20:30:49.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27340a62e0 con 0x7f27340a28c0 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272bfff640 4 set_mon_vals no callback set 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 monclient: dump: 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.390 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 10 client.24497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 10 monclient: _renew_subs 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f273411e9a0 con 0x7f27340a28c0 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 1 librados: init done 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f273406f830 0x7f273406fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f27340a6830 con 0x7f273406f830 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f273406f830 0x7f273406fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 10 monclient: get_auth_request con 0x7f273406f830 auth_method 0 2024-09-10T20:30:49.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f27300bb020 con 0x7f27340a28c0 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 10 monclient(hunting): handle_auth_done global_id 24500 payload 824 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f2739326640 1 -- 172.21.15.153:0/1528800753 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f27340a1f10 con 0x7f273406f830 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 1 --2- 172.21.15.153:0/1528800753 >> v2:172.21.15.176:6812/3003473398 conn(0x7f271803cea0 0x7f271803f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(206..206 src has 1..206) ==== 5531+0+0 (secure 0 0 0) 0x7f27300e96f0 con 0x7f27340a28c0 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 10 client.24497.objecter ms_dispatch 0x7f2734000ba0 osd_map(206..206 src has 1..206) 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 3 client.24497.objecter handle_osd_map got epochs [206,206] > 0 2024-09-10T20:30:49.392 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 3 client.24497.objecter handle_osd_map decoding full epoch 206 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f272a7fc640 20 client.24497.objecter dump_active .. 0 homeless 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273a328640 1 --2- 172.21.15.153:0/1528800753 >> v2:172.21.15.176:6812/3003473398 conn(0x7f271803cea0 0x7f271803f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.368+0000 7f273a328640 10 monclient: get_auth_request con 0x7f271803cea0 auth_method 0 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 10 client.24497.objecter ms_handle_connect 0x7f273406f830 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 10 client.24497.objecter resend_mon_ops 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f2739326640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f273406f830 msgr2=0x7f273406fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f2739326640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f273406f830 0x7f273406fc10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2724007270 tx=0x7f2724007730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2724017020 con 0x7f273406f830 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 10 monclient: dump: 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.393 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f272400fd00 con 0x7f273406f830 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:49.394 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f273406f830 msgr2=0x7f273406fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f273406f830 0x7f273406fc10 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2724007270 tx=0x7f2724007730 comp rx=0 tx=0).stop 2024-09-10T20:30:49.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f273406f830 msgr2=0x7f273406fc10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:49.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f273406f830 0x7f273406fc10 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2724007270 tx=0x7f2724007730 comp rx=0 tx=0).stop 2024-09-10T20:30:49.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f272a7fc640 10 client.24497.objecter ms_handle_connect 0x7f271803cea0 2024-09-10T20:30:49.395 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.372+0000 7f273a328640 1 --2- 172.21.15.153:0/1528800753 >> v2:172.21.15.176:6812/3003473398 conn(0x7f271803cea0 0x7f271803f360 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7f273411fd00 tx=0x7f272c00b040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f273bd8b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f273bd8b640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2734070150 0x7f27341d66f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f273bd8b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f273406a640 con 0x7f2734070150 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2734070150 0x7f27341d66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 10 monclient: get_auth_request con 0x7f2734070150 auth_method 0 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.397 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 10 monclient(hunting): handle_auth_done global_id 24503 payload 824 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.376+0000 7f2739326640 1 -- 172.21.15.153:0/1528800753 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f273411e270 con 0x7f2734070150 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f2739326640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2734070150 msgr2=0x7f27341d66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 10 client.24497.objecter ms_handle_connect 0x7f2734070150 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 10 client.24497.objecter resend_mon_ops 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f2739326640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2734070150 0x7f27341d66f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2724018000 tx=0x7f2724004950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2724017030 con 0x7f2734070150 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 10 monclient: dump: 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.398 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4720 (secure 0 0 0) 0x7f27240058c0 con 0x7f2734070150 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2734070150 msgr2=0x7f27341d66f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2734070150 0x7f27341d66f0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2724018000 tx=0x7f2724004950 comp rx=0 tx=0).stop 2024-09-10T20:30:49.399 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2734070150 msgr2=0x7f27341d66f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:49.400 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f272a7fc640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2734070150 0x7f27341d66f0 secure :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f2724018000 tx=0x7f2724004950 comp rx=0 tx=0).stop 2024-09-10T20:30:49.400 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:30:49.400 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:30:49.400 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:30:49.400 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:30:49.400 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 26, 2024-09-10T20:30:49.400 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 150949, 2024-09-10T20:30:49.400 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:30:49.400 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:30:49.401 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:30:49.401 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:30:49.401 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.401 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-10T20:30:49.401 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:49.402 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:49.403 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:49.404 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:49.405 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.406 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:30:49.407 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:30:49.408 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.409 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:49.410 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:49.411 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.412 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:30:49.413 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.414 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.414 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.414 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:30:49.414 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:30:49.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 10 librados: watch_flush enter 2024-09-10T20:30:49.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 10 librados: watch_flush exit 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 20 client.24497.objecter shutdown clearing up homeless session... 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 10 client.24497.objecter successfully canceled tick 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 >> v2:172.21.15.176:6812/3003473398 conn(0x7f271803cea0 msgr2=0x7f271803f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 --2- 172.21.15.153:0/1528800753 >> v2:172.21.15.176:6812/3003473398 conn(0x7f271803cea0 0x7f271803f360 secure :-1 s=READY pgs=3323 cs=0 l=1 rev1=1 crypto rx=0x7f273411fd00 tx=0x7f272c00b040 comp rx=0 tx=0).stop 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 10 monclient: shutdown 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 msgr2=0x7f2734072ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f2734072ee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f273001cbd0 tx=0x7f27300190d0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273a328640 1 -- 172.21.15.153:0/1528800753 reap_dead start 2024-09-10T20:30:49.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 shutdown_connections 2024-09-10T20:30:49.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 --2- 172.21.15.153:0/1528800753 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f27340a28c0 0x7f2734072ee0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 >> 172.21.15.153:0/1528800753 conn(0x7f27340a71b0 msgr2=0x7f273413c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:49.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 shutdown_connections 2024-09-10T20:30:49.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 -- 172.21.15.153:0/1528800753 wait complete. 2024-09-10T20:30:49.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.380+0000 7f273bd8b640 1 librados: shutdown 2024-09-10T20:30:49.417 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:30:49.417 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T20:30:49.417 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.528+0000 7fb9ff2b6640 10 monclient: get_monmap_and_config 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.528+0000 7fb9ff2b6640 10 monclient: build_initial_monmap 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.528+0000 7fb9ff2b6640 10 monclient: monmap: 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:30:49.533085+0000 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:30:49.533085+0000 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.535 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:30:49.536 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:30:49.536 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:30:49.536 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:30:49.536 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:30:49.536 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.528+0000 7fb9ff2b6640 1 Processor -- start 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 1 -- start start 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient: init 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient: _add_conns ranks=[0,3,2,1,4] 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb9f81364f0 addr v1:172.21.15.153:6789/0 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb9f81360a0 0x7fb9f813b420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f81360a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 0x7fb9f813d970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient(hunting): picked mon.noname-b con 0x7fb9f80a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9f80a24c0 con 0x7fb9f81364f0 2024-09-10T20:30:49.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fb9f80a27a0 con 0x7fb9f81360a0 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f80a2a80 con 0x7fb9f80a1990 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9ff2b6640 10 monclient(hunting): authenticate will time out at 1209.672661s 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fc851640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fb9f81360a0 0x7fb9f813b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 0x7fb9f813d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd052640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fb9f81364f0 0x7fb9f81368d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:44486/0 (socket says 172.21.15.153:44486) 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd052640 1 -- 172.21.15.153:0/3862147168 learned_addr learned my addr 172.21.15.153:0/3862147168 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fc851640 10 monclient(hunting): get_auth_request con 0x7fb9f81360a0 auth_method 0 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fc851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fc851640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.539 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fc851640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient(hunting): get_auth_request con 0x7fb9f80a1990 auth_method 0 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fc851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fc851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fc851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient(hunting): handle_auth_done global_id 24506 payload 274 2024-09-10T20:30:49.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 1 -- 172.21.15.153:0/3862147168 >> v2:172.21.15.153:3300/0 conn(0x7fb9f81360a0 msgr2=0x7fb9f813b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:49.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 1 --2- 172.21.15.153:0/3862147168 >> v2:172.21.15.153:3300/0 conn(0x7fb9f81360a0 0x7fb9f813b420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 1 -- 172.21.15.153:0/3862147168 >> v1:172.21.15.153:6789/0 conn(0x7fb9f81364f0 legacy=0x7fb9f81368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:30:49.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient: _finish_hunting 0 2024-09-10T20:30:49.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 1 monclient: found mon.noname-b 2024-09-10T20:30:49.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:49.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:30:49.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fd853640 1 -- 172.21.15.153:0/3862147168 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f80a2d60 con 0x7fb9f80a1990 2024-09-10T20:30:49.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.532+0000 7fb9fc851640 1 --2- 172.21.15.153:0/3862147168 >> v2:172.21.15.153:3300/0 conn(0x7fb9f81360a0 0x7fb9f813b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:30:49.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fd853640 1 -- 172.21.15.153:0/3862147168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 msgr2=0x7fb9f813d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fd853640 1 --2- 172.21.15.153:0/3862147168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 0x7fb9f813d970 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000f370 tx=0x7fb9f0020cc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=626481c4df76bd78 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 1 -- 172.21.15.153:0/3862147168 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f00a9b30 con 0x7fb9f80a1990 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 10 monclient: dump: 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.545 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 10 monclient: _finish_auth 0 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 10 monclient: _check_auth_tickets 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 1 -- 172.21.15.153:0/3862147168 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f00a9cd0 con 0x7fb9f80a1990 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 1 -- 172.21.15.153:0/3862147168 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f000d2f0 con 0x7fb9f80a1990 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9effff640 10 monclient: dump: 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.546 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 5 monclient: authenticate success, global_id 24506 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient: get_monmap_and_config success 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 4 set_mon_vals no callback set 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient: shutdown 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3862147168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 msgr2=0x7fb9f813d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 --2- 172.21.15.153:0/3862147168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 0x7fb9f813d970 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb9f000f370 tx=0x7fb9f0020cc0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3862147168 shutdown_connections 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 --2- 172.21.15.153:0/3862147168 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 0x7fb9f813d970 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 --2- 172.21.15.153:0/3862147168 >> v2:172.21.15.153:3300/0 conn(0x7fb9f81360a0 0x7fb9f813b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3862147168 >> 172.21.15.153:0/3862147168 conn(0x7fb9f80a71b0 msgr2=0x7fb9f80a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3862147168 shutdown_connections 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3862147168 wait complete. 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient: build_initial_monmap 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient: monmap: 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:30:49.548 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 librados: starting msgr at 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 librados: starting objecter 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 Processor -- start 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 -- start start 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 librados: setting wanted keys 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 librados: calling monclient init 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient: init 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient: _reopen_session rank -1 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:30:49.549 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 0x7fb9f81cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f80a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f81360a0 0x7fb9f813d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f81360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f81364f0 0x7fb9f813d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient(hunting): picked mon.noname-b con 0x7fb9f81364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f813eba0 con 0x7fb9f81360a0 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f813ea20 con 0x7fb9f81364f0 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fb9f813ed20 con 0x7fb9f80a1990 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient(hunting): _renew_subs 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9ff2b6640 10 monclient(hunting): authenticate will time out at 1209.679828s 2024-09-10T20:30:49.550 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fc851640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f81360a0 0x7fb9f813d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fd052640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 0x7fb9f81cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fc851640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f81360a0 0x7fb9f813d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:49808/0 (socket says 172.21.15.153:49808) 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fc851640 1 -- 172.21.15.153:0/3103591360 learned_addr learned my addr 172.21.15.153:0/3103591360 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fc851640 10 monclient(hunting): get_auth_request con 0x7fb9f81360a0 auth_method 0 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fc851640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fc851640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fc851640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fd052640 10 monclient(hunting): get_auth_request con 0x7fb9f80a1990 auth_method 0 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fd052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fd052640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.536+0000 7fb9fd052640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.551 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 10 monclient(hunting): handle_auth_done global_id 34158 payload 824 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 msgr2=0x7fb9f81cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fb9f80a1990 0x7fb9f81cbfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f81364f0 msgr2=0x7fb9f813d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f81364f0 0x7fb9f813d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 10 monclient: _finish_hunting 0 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 1 monclient: found mon.noname-a 2024-09-10T20:30:49.552 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:30:49.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:30:49.553 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 1 -- 172.21.15.153:0/3103591360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9f809e0a0 con 0x7fb9f81360a0 2024-09-10T20:30:49.555 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f81360a0 msgr2=0x7fb9f813d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f81360a0 0x7fb9f813d000 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400e9b0 tx=0x7fb9f400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 client.?.objecter ms_handle_connect 0x7fb9f81360a0 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4017070 con 0x7fb9f81360a0 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 monclient: dump: 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.556 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 monclient: _finish_auth 0 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 monclient: _check_auth_tickets 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9f400cb30 con 0x7fb9f81360a0 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 5 monclient: authenticate success, global_id 34158 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 10 monclient: _renew_subs 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:49.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9f813aae0 con 0x7fb9f81360a0 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9f4013470 con 0x7fb9f81360a0 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 10 monclient: dump: 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.558 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 10 client.34158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 10 monclient: _renew_subs 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ef7fe640 4 set_mon_vals no callback set 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9f813a800 con 0x7fb9f81360a0 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 1 librados: init done 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137900 0x7fb9f802d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f8130f00 con 0x7fb9f8137900 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137900 0x7fb9f802d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient: get_auth_request con 0x7fb9f8137900 auth_method 0 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fd052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9fc851640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f81360a0 0x7fb9f813d000 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400e9b0 tx=0x7fb9f400ee70 comp rx=0 tx=0).queue_received will delay after 1.726e+09 on 0x7fb9f404f840 osd_map(207..207 src has 1..207) 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.540+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fb9f401a070 con 0x7fb9f81360a0 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9fd052640 10 monclient(hunting): handle_auth_done global_id 20017 payload 824 2024-09-10T20:30:49.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9fd052640 1 -- 172.21.15.153:0/3103591360 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb9f81368d0 con 0x7fb9f8137900 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 1 --2- 172.21.15.153:0/3103591360 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb9dc03cea0 0x7fb9dc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9fd853640 1 --2- 172.21.15.153:0/3103591360 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb9dc03cea0 0x7fb9dc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9fd853640 10 monclient: get_auth_request con 0x7fb9dc03cea0 auth_method 0 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 10 client.34158.objecter ms_handle_connect 0x7fb9f8137900 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 10 client.34158.objecter resend_mon_ops 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9fd052640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137900 msgr2=0x7fb9f802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9fd052640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137900 0x7fb9f802d450 secure :-1 s=READY pgs=5649 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800eaa0 tx=0x7fb9e800eb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e800ee60 con 0x7fb9f8137900 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 10 monclient: dump: 2024-09-10T20:30:49.561 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 10 client.34158.objecter ms_handle_connect 0x7fb9dc03cea0 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9fd853640 1 --2- 172.21.15.153:0/3103591360 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb9dc03cea0 0x7fb9dc03f360 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00af000 tx=0x7fb9f0004ac0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb9e8019070 con 0x7fb9f8137900 2024-09-10T20:30:49.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:30:49.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:30:49.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137900 msgr2=0x7fb9f802d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137900 0x7fb9f802d450 secure :-1 s=READY pgs=5649 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800eaa0 tx=0x7fb9e800eb90 comp rx=0 tx=0).stop 2024-09-10T20:30:49.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137900 msgr2=0x7fb9f802d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:49.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.544+0000 7fb9edffb640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137900 0x7fb9f802d450 secure :-1 s=CLOSED pgs=5649 cs=0 l=1 rev1=1 crypto rx=0x7fb9e800eaa0 tx=0x7fb9e800eb90 comp rx=0 tx=0).stop 2024-09-10T20:30:49.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9ff2b6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:30:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9ff2b6640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137ce0 0x7fb9f802d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:30:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9ff2b6640 10 monclient(hunting): start opening mon connection 2024-09-10T20:30:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fb9f813da80 con 0x7fb9f8137ce0 2024-09-10T20:30:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137ce0 0x7fb9f802d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:30:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 10 monclient: get_auth_request con 0x7fb9f8137ce0 auth_method 0 2024-09-10T20:30:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:30:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:30:49.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 10 monclient(hunting): handle_auth_done global_id 20020 payload 824 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.552+0000 7fb9fd052640 1 -- 172.21.15.153:0/3103591360 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb9f8070f40 con 0x7fb9f8137ce0 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9fd052640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137ce0 msgr2=0x7fb9f802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9fd052640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137ce0 0x7fb9f802d990 secure :-1 s=READY pgs=5650 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80070c0 tx=0x7fb9e8004950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 10 client.34158.objecter ms_handle_connect 0x7fb9f8137ce0 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 10 client.34158.objecter resend_mon_ops 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9e801ebd0 con 0x7fb9f8137ce0 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:30:49.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 10 monclient: dump: 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7fb9e8014480 con 0x7fb9f8137ce0 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:30:49.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137ce0 msgr2=0x7fb9f802d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137ce0 0x7fb9f802d990 secure :-1 s=READY pgs=5650 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80070c0 tx=0x7fb9e8004950 comp rx=0 tx=0).stop 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137ce0 msgr2=0x7fb9f802d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fb9f8137ce0 0x7fb9f802d990 secure :-1 s=CLOSED pgs=5650 cs=0 l=1 rev1=1 crypto rx=0x7fb9e80070c0 tx=0x7fb9e8004950 comp rx=0 tx=0).stop 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 26, 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 296198, 2024-09-10T20:30:49.568 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:49.569 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:49.570 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:30:49.571 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:30:49.572 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:30:49.573 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.574 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.575 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:49.576 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.577 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.578 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:30:49.579 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 7 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:30:49.580 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:30:49.581 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:30:49.581 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:30:49.581 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:30:49.581 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:30:49.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 10 librados: watch_flush enter 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 10 librados: watch_flush exit 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 20 client.34158.objecter shutdown clearing up homeless session... 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 10 client.34158.objecter successfully canceled tick 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb9dc03cea0 msgr2=0x7fb9dc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 1 --2- 172.21.15.153:0/3103591360 >> v2:172.21.15.176:6812/3003473398 conn(0x7fb9dc03cea0 0x7fb9dc03f360 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7fb9f00af000 tx=0x7fb9f0004ac0 comp rx=0 tx=0).stop 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 10 monclient: shutdown 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f81360a0 msgr2=0x7fb9f813d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 1 --2- 172.21.15.153:0/3103591360 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fb9f81360a0 0x7fb9f813d000 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb9f400e9b0 tx=0x7fb9f400ee70 comp rx=0 tx=0).stop 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9fd853640 1 -- 172.21.15.153:0/3103591360 reap_dead start 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 1 -- 172.21.15.153:0/3103591360 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 5229+0+0 (unknown 0 0 0) 0x7fb9f404f840 con 0x7fb9f81360a0 2024-09-10T20:30:49.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 10 client.34158.objecter ms_dispatch 0x7fb9f8000ba0 osd_map(207..207 src has 1..207) 2024-09-10T20:30:49.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9edffb640 10 librados: disconnected, discarding osd_map(207..207 src has 1..207) 2024-09-10T20:30:49.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9fd853640 1 -- 172.21.15.153:0/3103591360 reap_dead start 2024-09-10T20:30:49.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 shutdown_connections 2024-09-10T20:30:49.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 >> 172.21.15.153:0/3103591360 conn(0x7fb9f80a71b0 msgr2=0x7fb9f80a5110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:30:49.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 shutdown_connections 2024-09-10T20:30:49.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 1 -- 172.21.15.153:0/3103591360 wait complete. 2024-09-10T20:30:49.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:30:49.556+0000 7fb9ff2b6640 1 librados: shutdown 2024-09-10T20:30:49.584 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:30:49.585 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T20:30:49.585 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-10T20:30:49.585 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T20:30:49.585 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T20:30:49.585 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T20:30:49.585 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:30:49.596 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T20:30:49.596 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-10T20:30:49.596 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-10T20:30:49.597 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T20:31:04.598 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-10T20:31:04.598 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-10T20:31:04.598 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T20:31:04.599 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:31:04.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient: get_monmap_and_config 2024-09-10T20:31:04.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient: build_initial_monmap 2024-09-10T20:31:04.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient: monmap: 2024-09-10T20:31:04.729 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:31:04.729 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:31:04.727477+0000 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:31:04.727477+0000 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:31:04.730 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:04.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 1 Processor -- start 2024-09-10T20:31:04.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 1 -- start start 2024-09-10T20:31:04.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient: init 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient: _reopen_session rank -1 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient: _add_conns ranks=[2,0,4,1,3] 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 0x7fd09c09d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fd09c09ce10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd09c0a0270 addr v1:172.21.15.153:6789/0 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fd09c09d730 addr v2:172.21.15.176:3300/0 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09c0a3040 con 0x7fd09c0a0270 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd09c09ea30 con 0x7fd09c09d730 2024-09-10T20:31:04.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c09f5b0 con 0x7fd09c09ce10 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient(hunting): _renew_subs 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd0a1f5d640 10 monclient(hunting): authenticate will time out at 1224.867659s 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd09b7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 msgr2=0x7fd09c09d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd09b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 0x7fd09c09d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd092ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd09c0a0270 0x7fd09c135580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:36428/0 (socket says 172.21.15.153:36428) 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.723+0000 7fd092ffd640 1 -- 172.21.15.153:0/1617571584 learned_addr learned my addr 172.21.15.153:0/1617571584 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd09bfff640 10 monclient(hunting): get_auth_request con 0x7fd09c09d730 auth_method 0 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd09bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd09bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:04.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd09bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:04.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd09bfff640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 msgr2=0x7fd09c135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:31:04.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd09bfff640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 msgr2=0x7fd09c135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:31:04.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:31:04.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:31:04.735 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.727+0000 7fd092ffd640 1 -- 172.21.15.153:0/1617571584 >> v1:172.21.15.153:6789/0 conn(0x7fd09c0a0270 legacy=0x7fd09c135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:04.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.927+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 msgr2=0x7fd09c09d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:31:04.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.927+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 0x7fd09c09d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:31:04.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.931+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:04.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.931+0000 7fd09bfff640 10 monclient(hunting): get_auth_request con 0x7fd09c09d730 auth_method 0 2024-09-10T20:31:04.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.931+0000 7fd09bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:04.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.931+0000 7fd09bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:04.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.931+0000 7fd09bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:31:04.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.931+0000 7fd09bfff640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 msgr2=0x7fd09c135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:31:04.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.931+0000 7fd09bfff640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 msgr2=0x7fd09c135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:31:04.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.931+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:31:04.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:04.931+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:31:05.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.331+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 msgr2=0x7fd09c09d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:31:05.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.331+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 0x7fd09c09d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:31:05.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.335+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:05.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.335+0000 7fd09bfff640 10 monclient(hunting): get_auth_request con 0x7fd09c09d730 auth_method 0 2024-09-10T20:31:05.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.335+0000 7fd09bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:05.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.335+0000 7fd09bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:05.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.335+0000 7fd09bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:31:05.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.335+0000 7fd09bfff640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 msgr2=0x7fd09c135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:31:05.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.335+0000 7fd09bfff640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 msgr2=0x7fd09c135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:31:05.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.335+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:31:05.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:05.335+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:31:06.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.135+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 msgr2=0x7fd09c09d1f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:31:06.140 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.135+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 0x7fd09c09d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:31:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.139+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.139+0000 7fd09bfff640 10 monclient(hunting): get_auth_request con 0x7fd09c09d730 auth_method 0 2024-09-10T20:31:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.139+0000 7fd09bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.139+0000 7fd09bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.139+0000 7fd09bfff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:31:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.139+0000 7fd09bfff640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 msgr2=0x7fd09c135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:31:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.139+0000 7fd09bfff640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 msgr2=0x7fd09c135c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:31:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.139+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:31:06.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:06.139+0000 7fd09bfff640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:31:07.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 10 monclient(hunting): tick 2024-09-10T20:31:07.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 10 monclient(hunting): _check_auth_tickets 2024-09-10T20:31:07.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 1 monclient(hunting): continuing hunt 2024-09-10T20:31:07.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T20:31:07.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 1 -- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 msgr2=0x7fd09c09d1f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:31:07.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd09c09ce10 0x7fd09c09d1f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:07.732 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 msgr2=0x7fd09c135c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.176:3300/0 conn(0x7fd09c09d730 0x7fd09c135c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 1 -- 172.21.15.153:0/1617571584 >> v1:172.21.15.153:6789/0 conn(0x7fd09c0a0270 legacy=0x7fd09c135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 10 monclient: _add_conns ranks=[3,1] 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 10 monclient(hunting): picked mon.noname-a con 0x7fd07c003770 addr v2:172.21.15.153:3300/0 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd07c0084d0 addr v1:172.21.15.176:6789/0 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 1 -- 172.21.15.153:0/1617571584 --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09c09f890 con 0x7fd07c0084d0 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 1 -- 172.21.15.153:0/1617571584 --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fd09c09ea30 con 0x7fd07c003770 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.723+0000 7fd099ffb640 10 monclient(hunting): _renew_subs 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request con 0x7fd07c003770 auth_method 0 2024-09-10T20:31:07.733 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd09b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd09b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 msgr2=0x7fd07c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:31:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 msgr2=0x7fd07c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:31:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:31:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:31:07.734 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.727+0000 7fd092ffd640 1 -- 172.21.15.153:0/1617571584 >> v1:172.21.15.176:6789/0 conn(0x7fd07c0084d0 legacy=0x7fd07c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:07.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.931+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:07.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.931+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request con 0x7fd07c003770 auth_method 0 2024-09-10T20:31:07.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.931+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:07.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.931+0000 7fd09b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:07.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.931+0000 7fd09b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:31:07.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.931+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 msgr2=0x7fd07c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:31:07.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.931+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 msgr2=0x7fd07c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:31:07.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.931+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:31:07.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:07.931+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:31:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:08.335+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:08.335+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request con 0x7fd07c003770 auth_method 0 2024-09-10T20:31:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:08.335+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:08.335+0000 7fd09b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:08.335+0000 7fd09b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:31:08.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:08.335+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 msgr2=0x7fd07c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:31:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:08.335+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 msgr2=0x7fd07c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:31:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:08.335+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:31:08.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:08.335+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:31:09.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.139+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:09.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.139+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request con 0x7fd07c003770 auth_method 0 2024-09-10T20:31:09.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.139+0000 7fd09b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:09.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.139+0000 7fd09b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:09.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.139+0000 7fd09b7fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:31:09.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.139+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 msgr2=0x7fd07c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:31:09.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.139+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 msgr2=0x7fd07c005bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:31:09.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.139+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:31:09.145 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.139+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:31:09.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 <== mon.1 v1:172.21.15.176:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2668443228 0 0) 0x7fd09c09f890 con 0x7fd07c0084d0 2024-09-10T20:31:09.658 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient(hunting): _negotiate 2024-09-10T20:31:09.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:09.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:09.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient(hunting): my global_id is 20023 2024-09-10T20:31:09.659 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 --> v1:172.21.15.176:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd088003530 con 0x7fd07c0084d0 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 <== mon.1 v1:172.21.15.176:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4199148282 0 0) 0x7fd088003530 con 0x7fd07c0084d0 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 msgr2=0x7fd07c005bd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 --2- 172.21.15.153:0/1617571584 >> v2:172.21.15.153:3300/0 conn(0x7fd07c003770 0x7fd07c005bd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient: _finish_hunting 0 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 monclient: found mon.noname-c-legacy 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient: _send_mon_message to mon.noname-c-legacy at v1:172.21.15.176:6789/0 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 --> v1:172.21.15.176:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd07c00d300 con 0x7fd07c0084d0 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient: _finish_auth 0 2024-09-10T20:31:09.661 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient: _check_auth_tickets 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 <== mon.1 v1:172.21.15.176:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3765011595 0 0) 0x7fd084001940 con 0x7fd07c0084d0 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient: got monmap 1 from mon.noname-c-legacy (according to old e1) 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient: dump: 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:09.662 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 monclient: mon.b has (v2) addrs [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] but i'm connected to v1:172.21.15.176:6789/0, reconnecting 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 10 monclient: _reopen_session rank -1 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.651+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 >> v1:172.21.15.176:6789/0 conn(0x7fd07c0084d0 legacy=0x7fd07c00a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09bfff640 1 -- 172.21.15.153:0/1617571584 reap_dead start 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 0x7fd088003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 10 monclient(hunting): picked mon.c con 0x7fd0880039c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd088004810 0x7fd088004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 10 monclient(hunting): picked mon.b con 0x7fd088004810 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0880052a0 0x7fd08800b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 10 monclient(hunting): picked mon.a con 0x7fd0880052a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:31:09.663 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c09ea30 con 0x7fd0880052a0 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd08800c1f0 con 0x7fd088004810 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd08800c3f0 con 0x7fd0880039c0 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 10 monclient(hunting): _renew_subs 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09affd640 10 monclient(hunting): _finish_auth 0 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09b7fe640 1 -- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 msgr2=0x7fd088003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 0x7fd088003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd088004810 0x7fd088004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient(hunting): get_auth_request con 0x7fd088004810 auth_method 0 2024-09-10T20:31:09.664 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient(hunting): handle_auth_done global_id 20023 payload 293 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 1 -- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 msgr2=0x7fd088003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 0x7fd088003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 1 -- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0880052a0 msgr2=0x7fd08800b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0880052a0 0x7fd08800b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 1 monclient: found mon.b 2024-09-10T20:31:09.665 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:31:09.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:31:09.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 1 -- 172.21.15.153:0/1617571584 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd088003530 con 0x7fd088004810 2024-09-10T20:31:09.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient: _finish_auth 0 2024-09-10T20:31:09.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:31:09.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd092ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:31:09.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd0a1f5d640 5 monclient: authenticate success, global_id 20023 2024-09-10T20:31:09.666 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.655+0000 7fd0a1f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:31:09.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd092ffd640 1 -- 172.21.15.153:0/1617571584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd088004810 msgr2=0x7fd088004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:09.668 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd092ffd640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd088004810 0x7fd088004c90 secure :-1 s=READY pgs=5657 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0a7e20 tx=0x7fd08401f2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aac7d1d81e26ecce server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd08400c0e0 con 0x7fd088004810 2024-09-10T20:31:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:31:09.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 10 monclient: dump: 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:31:09.670 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd084005ae0 con 0x7fd088004810 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 10 monclient: get_monmap_and_config success 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 4 set_mon_vals no callback set 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 1 -- 172.21.15.153:0/1617571584 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd084007070 con 0x7fd088004810 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd09affd640 10 monclient: dump: 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:09.671 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:09.672 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:09.672 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:09.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 10 monclient: shutdown 2024-09-10T20:31:09.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:31:09.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/1617571584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd088004810 msgr2=0x7fd088004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:09.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd088004810 0x7fd088004c90 secure :-1 s=READY pgs=5657 cs=0 l=1 rev1=1 crypto rx=0x7fd09c0a7e20 tx=0x7fd08401f2d0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/1617571584 shutdown_connections 2024-09-10T20:31:09.673 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd0880052a0 0x7fd08800b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd088004810 0x7fd088004c90 unknown :-1 s=CLOSED pgs=5657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/1617571584 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 0x7fd088003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/1617571584 >> 172.21.15.153:0/1617571584 conn(0x7fd09c0a71b0 msgr2=0x7fd09c0a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/1617571584 shutdown_connections 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.663+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/1617571584 wait complete. 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient: build_initial_monmap 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient: monmap: 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:09.674 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:31:09.675 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:31:09.675 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:31:09.675 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:09.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 librados: starting msgr at 2024-09-10T20:31:09.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 librados: starting objecter 2024-09-10T20:31:09.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 Processor -- start 2024-09-10T20:31:09.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 -- start start 2024-09-10T20:31:09.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 librados: setting wanted keys 2024-09-10T20:31:09.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 librados: calling monclient init 2024-09-10T20:31:09.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient: init 2024-09-10T20:31:09.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient: _reopen_session rank -1 2024-09-10T20:31:09.679 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 0x7fd09c120f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient(hunting): picked mon.noname-c con 0x7fd0880039c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd088004810 0x7fd09c11e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient(hunting): picked mon.noname-a con 0x7fd088004810 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0880052a0 0x7fd09c11eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd09b7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 msgr2=0x7fd09c120f70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient(hunting): picked mon.noname-b con 0x7fd0880052a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c0a2b20 con 0x7fd088004810 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:09.680 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd09c09db60 con 0x7fd0880052a0 2024-09-10T20:31:09.681 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd09b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 0x7fd09c120f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:31:09.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:09.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd09c0a2ca0 con 0x7fd0880039c0 2024-09-10T20:31:09.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient(hunting): _renew_subs 2024-09-10T20:31:09.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.667+0000 7fd0a1f5d640 10 monclient(hunting): authenticate will time out at 1229.811947s 2024-09-10T20:31:09.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd088004810 0x7fd09c11e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:09.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd088004810 0x7fd09c11e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34014/0 (socket says 172.21.15.153:34014) 2024-09-10T20:31:09.682 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 -- 172.21.15.153:0/694444126 learned_addr learned my addr 172.21.15.153:0/694444126 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:31:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient(hunting): get_auth_request con 0x7fd088004810 auth_method 0 2024-09-10T20:31:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd09bfff640 1 --2- 172.21.15.153:0/694444126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0880052a0 0x7fd09c11eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:09.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd09bfff640 10 monclient(hunting): get_auth_request con 0x7fd0880052a0 auth_method 0 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd09bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd09bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd09bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd09bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd09bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd09bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient(hunting): handle_auth_done global_id 34161 payload 824 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 -- 172.21.15.153:0/694444126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 msgr2=0x7fd09c120f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 --2- 172.21.15.153:0/694444126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 0x7fd09c120f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 -- 172.21.15.153:0/694444126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0880052a0 msgr2=0x7fd09c11eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 --2- 172.21.15.153:0/694444126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0880052a0 0x7fd09c11eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 monclient: found mon.noname-a 2024-09-10T20:31:09.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 -- 172.21.15.153:0/694444126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09c122770 con 0x7fd088004810 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 -- 172.21.15.153:0/694444126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd088004810 msgr2=0x7fd09c11e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd092ffd640 1 --2- 172.21.15.153:0/694444126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd088004810 0x7fd09c11e9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd08401f310 tx=0x7fd088005840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 client.?.objecter ms_handle_connect 0x7fd088004810 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 1 -- 172.21.15.153:0/694444126 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0840076d0 con 0x7fd088004810 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 monclient: dump: 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.685 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 monclient: _finish_auth 0 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 monclient: _check_auth_tickets 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:31:09.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 1 -- 172.21.15.153:0/694444126 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0840b6070 con 0x7fd088004810 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 1 -- 172.21.15.153:0/694444126 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0840b0430 con 0x7fd088004810 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd098ff9640 10 monclient: dump: 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:09.687 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.671+0000 7fd09a7fc640 4 set_mon_vals no callback set 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 5 monclient: authenticate success, global_id 34161 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 10 monclient: _renew_subs 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd09c073db0 con 0x7fd088004810 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 10 client.34161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 10 monclient: _renew_subs 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd09c121900 con 0x7fd088004810 2024-09-10T20:31:09.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 1 librados: init done 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd09c09d1f0 con 0x7fd088004810 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd098ff9640 1 -- 172.21.15.153:0/694444126 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fd0840b8020 con 0x7fd088004810 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd098ff9640 1 --2- 172.21.15.153:0/694444126 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd07c03d7d0 0x7fd07c03fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd098ff9640 1 -- 172.21.15.153:0/694444126 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 5229+0+0 (secure 0 0 0) 0x7fd0840ecb50 con 0x7fd088004810 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd098ff9640 10 client.34161.objecter ms_dispatch 0x7fd09c000ba0 osd_map(207..207 src has 1..207) 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd098ff9640 3 client.34161.objecter handle_osd_map got epochs [207,207] > 0 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd098ff9640 3 client.34161.objecter handle_osd_map decoding full epoch 207 2024-09-10T20:31:09.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd098ff9640 20 client.34161.objecter dump_active .. 0 homeless 2024-09-10T20:31:09.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/694444126 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd07c03d7d0 0x7fd07c03fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:09.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.675+0000 7fd09b7fe640 10 monclient: get_auth_request con 0x7fd07c03d7d0 auth_method 0 2024-09-10T20:31:09.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.679+0000 7fd09b7fe640 1 --2- 172.21.15.153:0/694444126 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd07c03d7d0 0x7fd07c03fc90 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7fd08c003990 tx=0x7fd08c00d040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:09.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.679+0000 7fd098ff9640 10 client.34161.objecter ms_handle_connect 0x7fd07c03d7d0 2024-09-10T20:31:09.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.679+0000 7fd098ff9640 1 -- 172.21.15.153:0/694444126 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fd084019080 con 0x7fd088004810 2024-09-10T20:31:09.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.679+0000 7fd098ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:31:09.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.679+0000 7fd098ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:31:09.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.907+0000 7fd0a1f5d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:31:09.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.907+0000 7fd0a1f5d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:31:09.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.907+0000 7fd0a1f5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:31:09.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.907+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd09c06a660 con 0x7fd088004810 2024-09-10T20:31:09.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.907+0000 7fd098ff9640 1 -- 172.21.15.153:0/694444126 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fd0840ece20 con 0x7fd088004810 2024-09-10T20:31:09.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.907+0000 7fd098ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:31:09.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.907+0000 7fd098ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:31:09.914 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":28,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:31:09.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 10 librados: watch_flush enter 2024-09-10T20:31:09.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 10 librados: watch_flush exit 2024-09-10T20:31:09.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 20 client.34161.objecter shutdown clearing up homeless session... 2024-09-10T20:31:09.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 10 client.34161.objecter successfully canceled tick 2024-09-10T20:31:09.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd07c03d7d0 msgr2=0x7fd07c03fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:09.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/694444126 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd07c03d7d0 0x7fd07c03fc90 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7fd08c003990 tx=0x7fd08c00d040 comp rx=0 tx=0).stop 2024-09-10T20:31:09.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 10 monclient: shutdown 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd088004810 msgr2=0x7fd09c11e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/694444126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd088004810 0x7fd09c11e9a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd08401f310 tx=0x7fd088005840 comp rx=0 tx=0).stop 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 shutdown_connections 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/694444126 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd0880052a0 0x7fd09c11eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/694444126 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd088004810 0x7fd09c11e9a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/694444126 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd07c03d7d0 0x7fd07c03fc90 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 --2- 172.21.15.153:0/694444126 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd0880039c0 0x7fd09c120f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 >> 172.21.15.153:0/694444126 conn(0x7fd09c0a71b0 msgr2=0x7fd09c074150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 shutdown_connections 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 -- 172.21.15.153:0/694444126 wait complete. 2024-09-10T20:31:09.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:09.911+0000 7fd0a1f5d640 1 librados: shutdown 2024-09-10T20:31:09.931 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T20:31:09.931 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T20:31:09.932 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:31:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.035+0000 7fba4a72b640 10 monclient: get_monmap_and_config 2024-09-10T20:31:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.035+0000 7fba4a72b640 10 monclient: build_initial_monmap 2024-09-10T20:31:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.035+0000 7fba4a72b640 10 monclient: monmap: 2024-09-10T20:31:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:31:10.043 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:31:10.041623+0000 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:31:10.041623+0000 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:31:10.044 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 1 Processor -- start 2024-09-10T20:31:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 1 -- start start 2024-09-10T20:31:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient: init 2024-09-10T20:31:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient: _reopen_session rank -1 2024-09-10T20:31:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient: _add_conns ranks=[4,1,3,2,0] 2024-09-10T20:31:10.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fba440a28c0 0x7fba4409d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient(hunting): picked mon.noname-c con 0x7fba440a28c0 addr v2:172.21.15.176:3300/0 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fba440a3060 addr v1:172.21.15.176:6789/0 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fba4413a7c0 0x7fba4413cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient(hunting): picked mon.noname-a con 0x7fba4413a7c0 addr v2:172.21.15.153:3300/0 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba4409f890 con 0x7fba440a3060 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fba440a34e0 con 0x7fba4413a7c0 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fba440a2ca0 con 0x7fba440a28c0 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient(hunting): _renew_subs 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba4a72b640 10 monclient(hunting): authenticate will time out at 1230.181286s 2024-09-10T20:31:10.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fba4413a7c0 0x7fba4413cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fba4413a7c0 0x7fba4413cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34024/0 (socket says 172.21.15.153:34024) 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 1 -- 172.21.15.153:0/2194617944 learned_addr learned my addr 172.21.15.153:0/2194617944 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient(hunting): get_auth_request con 0x7fba4413a7c0 auth_method 0 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:10.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient(hunting): handle_auth_done global_id 34164 payload 274 2024-09-10T20:31:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 1 -- 172.21.15.153:0/2194617944 >> v2:172.21.15.176:3300/0 conn(0x7fba440a28c0 msgr2=0x7fba4409d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:31:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 1 --2- 172.21.15.153:0/2194617944 >> v2:172.21.15.176:3300/0 conn(0x7fba440a28c0 0x7fba4409d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 1 -- 172.21.15.153:0/2194617944 >> v1:172.21.15.176:6789/0 conn(0x7fba440a3060 legacy=0x7fba4409d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:31:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient: _finish_hunting 0 2024-09-10T20:31:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 1 monclient: found mon.noname-a 2024-09-10T20:31:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:31:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:31:10.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.039+0000 7fba48cc8640 1 -- 172.21.15.153:0/2194617944 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4409e350 con 0x7fba4413a7c0 2024-09-10T20:31:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba48cc8640 1 -- 172.21.15.153:0/2194617944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4413a7c0 msgr2=0x7fba4413cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba48cc8640 1 --2- 172.21.15.153:0/2194617944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4413a7c0 0x7fba4413cbb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fba4406bfc0 tx=0x7fba3c01c3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6412e55f98d7db00 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 1 -- 172.21.15.153:0/2194617944 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0b0070 con 0x7fba4413a7c0 2024-09-10T20:31:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:31:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 10 monclient: dump: 2024-09-10T20:31:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.053 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 10 monclient: _finish_auth 0 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 5 monclient: authenticate success, global_id 34164 2024-09-10T20:31:10.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 1 -- 172.21.15.153:0/2194617944 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba3c0ab600 con 0x7fba4413a7c0 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient: get_monmap_and_config success 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 4 set_mon_vals no callback set 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 1 -- 172.21.15.153:0/2194617944 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c0ab9d0 con 0x7fba4413a7c0 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba42ffd640 10 monclient: dump: 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.055 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient: shutdown 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 -- 172.21.15.153:0/2194617944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4413a7c0 msgr2=0x7fba4413cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 --2- 172.21.15.153:0/2194617944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4413a7c0 0x7fba4413cbb0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fba4406bfc0 tx=0x7fba3c01c3b0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 -- 172.21.15.153:0/2194617944 shutdown_connections 2024-09-10T20:31:10.056 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 --2- 172.21.15.153:0/2194617944 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4413a7c0 0x7fba4413cbb0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 --2- 172.21.15.153:0/2194617944 >> v2:172.21.15.176:3300/0 conn(0x7fba440a28c0 0x7fba4409d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 -- 172.21.15.153:0/2194617944 >> 172.21.15.153:0/2194617944 conn(0x7fba440a71b0 msgr2=0x7fba440a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 -- 172.21.15.153:0/2194617944 shutdown_connections 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 -- 172.21.15.153:0/2194617944 wait complete. 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient: build_initial_monmap 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient: monmap: 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.057 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 librados: starting msgr at 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 librados: starting objecter 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 Processor -- start 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 -- start start 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 librados: setting wanted keys 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 librados: calling monclient init 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient: init 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient: _reopen_session rank -1 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba440a28c0 0x7fba441d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.058 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient(hunting): picked mon.noname-c con 0x7fba440a28c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba440a3060 0x7fba44135990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient(hunting): picked mon.noname-a con 0x7fba440a3060 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba4413a7c0 0x7fba44135ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient(hunting): picked mon.noname-b con 0x7fba4413a7c0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.043+0000 7fba4a72b640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba4413fa90 con 0x7fba440a3060 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fba4413f910 con 0x7fba4413a7c0 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fba4413fc10 con 0x7fba440a28c0 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 monclient(hunting): _renew_subs 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 monclient(hunting): authenticate will time out at 1230.188019s 2024-09-10T20:31:10.059 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba440a28c0 msgr2=0x7fba441d00f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba440a28c0 0x7fba441d00f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba440a3060 0x7fba44135990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba440a3060 0x7fba44135990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34026/0 (socket says 172.21.15.153:34026) 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 -- 172.21.15.153:0/2919424228 learned_addr learned my addr 172.21.15.153:0/2919424228 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient(hunting): get_auth_request con 0x7fba440a3060 auth_method 0 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba48cc8640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba4413a7c0 0x7fba44135ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.060 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient(hunting): handle_auth_done global_id 34167 payload 824 2024-09-10T20:31:10.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba440a28c0 msgr2=0x7fba441d00f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:31:10.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fba440a28c0 0x7fba441d00f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba4413a7c0 msgr2=0x7fba44135ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fba4413a7c0 0x7fba44135ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient: _finish_hunting 0 2024-09-10T20:31:10.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 monclient: found mon.noname-a 2024-09-10T20:31:10.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:31:10.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:31:10.061 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 -- 172.21.15.153:0/2919424228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba44120ff0 con 0x7fba440a3060 2024-09-10T20:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba440a3060 msgr2=0x7fba44135990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba440a3060 0x7fba44135990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00c490 tx=0x7fba2c001e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 client.?.objecter ms_handle_connect 0x7fba440a3060 2024-09-10T20:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c017070 con 0x7fba440a3060 2024-09-10T20:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba437fe640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba440a3060 0x7fba44135990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00c490 tx=0x7fba2c001e80 comp rx=0 tx=0).queue_received will delay after 1.726e+09 on 0x7fba2c003c10 mon_map magic: 0 2024-09-10T20:31:10.065 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 monclient: dump: 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 monclient: _finish_auth 0 2024-09-10T20:31:10.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 monclient: _check_auth_tickets 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 5 monclient: authenticate success, global_id 34167 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba2c003820 con 0x7fba440a3060 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 monclient: _renew_subs 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4409e0a0 con 0x7fba440a3060 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba427fc640 4 set_mon_vals no callback set 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 client.34167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 monclient: _renew_subs 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba44122ab0 con 0x7fba440a3060 2024-09-10T20:31:10.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 1 librados: init done 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4411fcf0 0x7fba4402d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba441d0690 con 0x7fba4411fcf0 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4411fcf0 0x7fba4402d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 10 monclient: get_auth_request con 0x7fba4411fcf0 auth_method 0 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:10.068 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba2c003c10 con 0x7fba440a3060 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 10 monclient: dump: 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.069 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fba2c01a360 con 0x7fba440a3060 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 10 monclient(hunting): handle_auth_done global_id 34170 payload 824 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.047+0000 7fba43fff640 1 -- 172.21.15.153:0/2919424228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fba4409d030 con 0x7fba4411fcf0 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 1 --2- 172.21.15.153:0/2919424228 >> v2:172.21.15.176:6812/3003473398 conn(0x7fba2403ce50 0x7fba2403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba48cc8640 1 --2- 172.21.15.153:0/2919424228 >> v2:172.21.15.176:6812/3003473398 conn(0x7fba2403ce50 0x7fba2403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba48cc8640 10 monclient: get_auth_request con 0x7fba2403ce50 auth_method 0 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba43fff640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4411fcf0 msgr2=0x7fba4402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 10 client.34167.objecter ms_handle_connect 0x7fba4411fcf0 2024-09-10T20:31:10.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 10 client.34167.objecter resend_mon_ops 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba43fff640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4411fcf0 0x7fba4402d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fba34002240 tx=0x7fba3400cef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34017070 con 0x7fba4411fcf0 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 10 monclient: dump: 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.071 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fba3400abf0 con 0x7fba4411fcf0 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4411fcf0 msgr2=0x7fba4402d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4411fcf0 0x7fba4402d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fba34002240 tx=0x7fba3400cef0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4411fcf0 msgr2=0x7fba4402d450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba4411fcf0 0x7fba4402d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba40ff9640 10 client.34167.objecter ms_handle_connect 0x7fba2403ce50 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.051+0000 7fba48cc8640 1 --2- 172.21.15.153:0/2919424228 >> v2:172.21.15.176:6812/3003473398 conn(0x7fba2403ce50 0x7fba2403f310 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0ab000 tx=0x7fba3c008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba4a72b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:31:10.072 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba4a72b640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba441200d0 0x7fba4402d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba4a72b640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fba4413c460 con 0x7fba441200d0 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba441200d0 0x7fba4402d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 10 monclient: get_auth_request con 0x7fba441200d0 auth_method 0 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 10 monclient(hunting): handle_auth_done global_id 34173 payload 824 2024-09-10T20:31:10.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 1 -- 172.21.15.153:0/2919424228 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fba4406a640 con 0x7fba441200d0 2024-09-10T20:31:10.074 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba441200d0 msgr2=0x7fba4402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba43fff640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba441200d0 0x7fba4402d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba34002240 tx=0x7fba34001310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 10 client.34167.objecter ms_handle_connect 0x7fba441200d0 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 10 client.34167.objecter resend_mon_ops 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba34017040 con 0x7fba441200d0 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 10 monclient: dump: 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.075 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4710 (secure 0 0 0) 0x7fba34007070 con 0x7fba441200d0 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba441200d0 msgr2=0x7fba4402d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba441200d0 0x7fba4402d990 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba34002240 tx=0x7fba34001310 comp rx=0 tx=0).stop 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba441200d0 msgr2=0x7fba4402d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:31:10.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba40ff9640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba441200d0 0x7fba4402d990 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fba34002240 tx=0x7fba34001310 comp rx=0 tx=0).stop 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 28, 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 30619, 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:31:10.077 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:31:10.078 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:31:10.079 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:31:10.080 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:31:10.081 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:31:10.082 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.083 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:31:10.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:31:10.083 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.083 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.083 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.083 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:31:10.083 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:31:10.083 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.083 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:31:10.084 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.085 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:31:10.086 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.087 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:31:10.088 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:31:10.089 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.090 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.090 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.090 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:31:10.090 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:31:10.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.059+0000 7fba4a72b640 10 librados: watch_flush enter 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 10 librados: watch_flush exit 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 20 client.34167.objecter shutdown clearing up homeless session... 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 10 client.34167.objecter successfully canceled tick 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 >> v2:172.21.15.176:6812/3003473398 conn(0x7fba2403ce50 msgr2=0x7fba2403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 1 --2- 172.21.15.153:0/2919424228 >> v2:172.21.15.176:6812/3003473398 conn(0x7fba2403ce50 0x7fba2403f310 secure :-1 s=READY pgs=3326 cs=0 l=1 rev1=1 crypto rx=0x7fba3c0ab000 tx=0x7fba3c008040 comp rx=0 tx=0).stop 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 10 monclient: shutdown 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba440a3060 msgr2=0x7fba44135990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 1 --2- 172.21.15.153:0/2919424228 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fba440a3060 0x7fba44135990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fba2c00c490 tx=0x7fba2c001e80 comp rx=0 tx=0).stop 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba48cc8640 1 -- 172.21.15.153:0/2919424228 reap_dead start 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba40ff9640 1 -- 172.21.15.153:0/2919424228 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 5229+0+0 (unknown 0 0 0) 0x7fba2c04ed50 con 0x7fba440a3060 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba40ff9640 10 client.34167.objecter ms_dispatch 0x7fba44000ba0 osd_map(207..207 src has 1..207) 2024-09-10T20:31:10.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba40ff9640 10 librados: disconnected, discarding osd_map(207..207 src has 1..207) 2024-09-10T20:31:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba48cc8640 1 -- 172.21.15.153:0/2919424228 reap_dead start 2024-09-10T20:31:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 shutdown_connections 2024-09-10T20:31:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 >> 172.21.15.153:0/2919424228 conn(0x7fba440a71b0 msgr2=0x7fba44074100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:31:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 shutdown_connections 2024-09-10T20:31:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 1 -- 172.21.15.153:0/2919424228 wait complete. 2024-09-10T20:31:10.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.063+0000 7fba4a72b640 1 librados: shutdown 2024-09-10T20:31:10.093 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T20:31:10.093 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T20:31:10.093 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:31:10.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.199+0000 7f02bece9640 10 monclient: get_monmap_and_config 2024-09-10T20:31:10.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.199+0000 7f02bece9640 10 monclient: build_initial_monmap 2024-09-10T20:31:10.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.199+0000 7f02bece9640 10 monclient: monmap: 2024-09-10T20:31:10.206 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:31:10.206 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.206 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:31:10.204377+0000 2024-09-10T20:31:10.206 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:31:10.204377+0000 2024-09-10T20:31:10.206 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:31:10.206 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.207 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:31:10.207 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:31:10.207 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:31:10.207 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:31:10.207 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:31:10.207 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.199+0000 7f02bece9640 1 Processor -- start 2024-09-10T20:31:10.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 1 -- start start 2024-09-10T20:31:10.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient: init 2024-09-10T20:31:10.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient: _reopen_session rank -1 2024-09-10T20:31:10.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient: _add_conns ranks=[3,1,2,4,0] 2024-09-10T20:31:10.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f02b80a41a0 0x7f02b80a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient(hunting): picked mon.noname-a con 0x7f02b80a41a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f02b80a4580 addr v1:172.21.15.176:6789/0 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a49d0 0x7f02b80a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient(hunting): picked mon.noname-b con 0x7f02b80a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02b80a6a20 con 0x7f02b80a4580 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f02b80a2570 con 0x7f02b80a41a0 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f02b8070400 con 0x7f02b80a49d0 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient(hunting): _renew_subs 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bece9640 10 monclient(hunting): authenticate will time out at 1230.344702s 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bd286640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a49d0 msgr2=0x7f02b80a60e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bd286640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a49d0 0x7f02b80a60e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:31:10.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f02b80a41a0 0x7f02b80a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f02b80a41a0 0x7f02b80a3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34068/0 (socket says 172.21.15.153:34068) 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 1 -- 172.21.15.153:0/1031032974 learned_addr learned my addr 172.21.15.153:0/1031032974 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient(hunting): get_auth_request con 0x7f02b80a41a0 auth_method 0 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient(hunting): handle_auth_done global_id 34176 payload 274 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 1 -- 172.21.15.153:0/1031032974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a49d0 msgr2=0x7f02b80a60e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 1 --2- 172.21.15.153:0/1031032974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a49d0 0x7f02b80a60e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 1 -- 172.21.15.153:0/1031032974 >> v1:172.21.15.176:6789/0 conn(0x7f02b80a4580 legacy=0x7f02b80a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:31:10.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient: _finish_hunting 0 2024-09-10T20:31:10.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 1 monclient: found mon.noname-a 2024-09-10T20:31:10.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:31:10.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:31:10.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.203+0000 7f02bca85640 1 -- 172.21.15.153:0/1031032974 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b80a3c80 con 0x7f02b80a41a0 2024-09-10T20:31:10.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bca85640 1 -- 172.21.15.153:0/1031032974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a41a0 msgr2=0x7f02b80a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:10.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bca85640 1 --2- 172.21.15.153:0/1031032974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a41a0 0x7f02b80a3740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009870 tx=0x7f02a801c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfd4cebb5976bc5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 1 -- 172.21.15.153:0/1031032974 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a80b0070 con 0x7f02b80a41a0 2024-09-10T20:31:10.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:31:10.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 10 monclient: dump: 2024-09-10T20:31:10.215 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.215 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.215 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.216 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.216 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.216 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.216 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.216 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.216 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.216 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 10 monclient: _finish_auth 0 2024-09-10T20:31:10.216 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 10 monclient: _check_auth_tickets 2024-09-10T20:31:10.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:31:10.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 1 -- 172.21.15.153:0/1031032974 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02a80ab600 con 0x7f02b80a41a0 2024-09-10T20:31:10.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:31:10.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 1 -- 172.21.15.153:0/1031032974 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a80ab9d0 con 0x7f02b80a41a0 2024-09-10T20:31:10.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 5 monclient: authenticate success, global_id 34176 2024-09-10T20:31:10.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 10 monclient: get_monmap_and_config success 2024-09-10T20:31:10.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 4 set_mon_vals no callback set 2024-09-10T20:31:10.217 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02b77fe640 10 monclient: dump: 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.218 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 10 monclient: shutdown 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 -- 172.21.15.153:0/1031032974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a41a0 msgr2=0x7f02b80a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 --2- 172.21.15.153:0/1031032974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a41a0 0x7f02b80a3740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f02a8009870 tx=0x7f02a801c440 comp rx=0 tx=0).stop 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 -- 172.21.15.153:0/1031032974 shutdown_connections 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 --2- 172.21.15.153:0/1031032974 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a49d0 0x7f02b80a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 --2- 172.21.15.153:0/1031032974 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a41a0 0x7f02b80a3740 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 -- 172.21.15.153:0/1031032974 >> 172.21.15.153:0/1031032974 conn(0x7f02b80ab7e0 msgr2=0x7f02b80a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 -- 172.21.15.153:0/1031032974 shutdown_connections 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 -- 172.21.15.153:0/1031032974 wait complete. 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 10 monclient: build_initial_monmap 2024-09-10T20:31:10.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 10 monclient: monmap: 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 librados: starting msgr at 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 librados: starting objecter 2024-09-10T20:31:10.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 Processor -- start 2024-09-10T20:31:10.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 -- start start 2024-09-10T20:31:10.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 librados: setting wanted keys 2024-09-10T20:31:10.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 1 librados: calling monclient init 2024-09-10T20:31:10.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.207+0000 7f02bece9640 10 monclient: init 2024-09-10T20:31:10.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient: _reopen_session rank -1 2024-09-10T20:31:10.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:31:10.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a41a0 0x7f02b8125d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient(hunting): picked mon.noname-c con 0x7f02b80a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:31:10.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a4580 0x7f02b8123c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient(hunting): picked mon.noname-a con 0x7f02b80a4580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:31:10.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b80a49d0 0x7f02b813cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient(hunting): picked mon.noname-b con 0x7f02b80a49d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:31:10.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f02b8143210 con 0x7f02b80a4580 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f02b80a6830 con 0x7f02b80a49d0 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f02b8143390 con 0x7f02b80a41a0 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient(hunting): _renew_subs 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient(hunting): authenticate will time out at 1230.352112s 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bca85640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a41a0 msgr2=0x7f02b8125d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bca85640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a41a0 0x7f02b8125d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a4580 0x7f02b8123c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a4580 0x7f02b8123c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:34078/0 (socket says 172.21.15.153:34078) 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 -- 172.21.15.153:0/2103119620 learned_addr learned my addr 172.21.15.153:0/2103119620 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient(hunting): get_auth_request con 0x7f02b80a4580 auth_method 0 2024-09-10T20:31:10.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient(hunting): handle_auth_done global_id 34179 payload 824 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bd286640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b80a49d0 0x7f02b813cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a41a0 msgr2=0x7f02b8125d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f02b80a41a0 0x7f02b8125d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b80a49d0 msgr2=0x7f02b813cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b80a49d0 0x7f02b813cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient: _finish_hunting 0 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 monclient: found mon.noname-a 2024-09-10T20:31:10.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 -- 172.21.15.153:0/2103119620 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02b8125200 con 0x7f02b80a4580 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a4580 msgr2=0x7f02b8123c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b7fff640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a4580 0x7f02b8123c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00c450 tx=0x7f02ac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 client.?.objecter ms_handle_connect 0x7f02b80a4580 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac0119a0 con 0x7f02b80a4580 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 monclient: dump: 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.228 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 monclient: _finish_auth 0 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 monclient: _check_auth_tickets 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 5 monclient: authenticate success, global_id 34179 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient: _renew_subs 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f02ac017070 con 0x7f02b80a4580 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02b8127560 con 0x7f02b80a4580 2024-09-10T20:31:10.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ac011da0 con 0x7f02b80a4580 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 10 monclient: dump: 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.230 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b6ffd640 4 set_mon_vals no callback set 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 client.34179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient: _renew_subs 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02b813d010 con 0x7f02b80a4580 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 librados: init done 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 0x7f02b81d0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f02b80a21c0 con 0x7f02b81d08e0 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.211+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f02ac01c460 con 0x7f02b80a4580 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bca85640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 0x7f02b81d0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bca85640 10 monclient: get_auth_request con 0x7f02b81d08e0 auth_method 0 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:10.231 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bca85640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bca85640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02b57fa640 1 --2- 172.21.15.153:0/2103119620 >> v2:172.21.15.176:6812/3003473398 conn(0x7f02a403cea0 0x7f02a403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(207..207 src has 1..207) ==== 5229+0+0 (secure 0 0 0) 0x7f02ac04f830 con 0x7f02b80a4580 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bca85640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 msgr2=0x7f02b81d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02b57fa640 10 client.34179.objecter ms_dispatch 0x7f02b8000ba0 osd_map(207..207 src has 1..207) 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bca85640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 msgr2=0x7f02b81d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02b57fa640 3 client.34179.objecter handle_osd_map got epochs [207,207] > 0 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02b57fa640 3 client.34179.objecter handle_osd_map decoding full epoch 207 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bca85640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 0x7f02b81d0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bca85640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 0x7f02b81d0cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02b57fa640 20 client.34179.objecter dump_active .. 0 homeless 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bd286640 1 --2- 172.21.15.153:0/2103119620 >> v2:172.21.15.176:6812/3003473398 conn(0x7f02a403cea0 0x7f02a403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bd286640 10 monclient: get_auth_request con 0x7f02a403cea0 auth_method 0 2024-09-10T20:31:10.232 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02b57fa640 10 client.34179.objecter ms_handle_connect 0x7f02a403cea0 2024-09-10T20:31:10.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.215+0000 7f02bd286640 1 --2- 172.21.15.153:0/2103119620 >> v2:172.21.15.176:6812/3003473398 conn(0x7f02a403cea0 0x7f02a403f360 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f02b8125510 tx=0x7f02b00071c0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 0x7f02b81d0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 10 monclient: get_auth_request con 0x7f02b81d08e0 auth_method 0 2024-09-10T20:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:10.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 10 monclient(hunting): handle_auth_done global_id 20032 payload 824 2024-09-10T20:31:10.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.419+0000 7f02bca85640 1 -- 172.21.15.153:0/2103119620 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f02b80a3740 con 0x7f02b81d08e0 2024-09-10T20:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02bca85640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 msgr2=0x7f02b81d0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:10.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 10 client.34179.objecter ms_handle_connect 0x7f02b81d08e0 2024-09-10T20:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02bca85640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 0x7f02b81d0cc0 secure :-1 s=READY pgs=5659 cs=0 l=1 rev1=1 crypto rx=0x7f02a80ad000 tx=0x7f02a80191e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 10 client.34179.objecter resend_mon_ops 2024-09-10T20:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a80b0040 con 0x7f02b81d08e0 2024-09-10T20:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 10 monclient: dump: 2024-09-10T20:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.429 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.430 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.430 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f02a80ab9b0 con 0x7f02b81d08e0 2024-09-10T20:31:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:31:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:31:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 msgr2=0x7f02b81d0cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 0x7f02b81d0cc0 secure :-1 s=READY pgs=5659 cs=0 l=1 rev1=1 crypto rx=0x7f02a80ad000 tx=0x7f02a80191e0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 msgr2=0x7f02b81d0cc0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:31:10.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.423+0000 7f02b57fa640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d08e0 0x7f02b81d0cc0 secure :-1 s=CLOSED pgs=5659 cs=0 l=1 rev1=1 crypto rx=0x7f02a80ad000 tx=0x7f02a80191e0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bece9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:31:10.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bece9640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d1200 0x7f02b809e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:31:10.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bece9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:31:10.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f02b813d300 con 0x7f02b81d1200 2024-09-10T20:31:10.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d1200 0x7f02b809e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:31:10.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 10 monclient: get_auth_request con 0x7f02b81d1200 auth_method 0 2024-09-10T20:31:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:31:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:31:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:31:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:31:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:31:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:31:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 10 monclient(hunting): handle_auth_done global_id 20035 payload 824 2024-09-10T20:31:10.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.431+0000 7f02bca85640 1 -- 172.21.15.153:0/2103119620 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f02b806cd40 con 0x7f02b81d1200 2024-09-10T20:31:10.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bca85640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d1200 msgr2=0x7f02b809e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:31:10.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 10 client.34179.objecter ms_handle_connect 0x7f02b81d1200 2024-09-10T20:31:10.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 10 client.34179.objecter resend_mon_ops 2024-09-10T20:31:10.441 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bca85640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d1200 0x7f02b809e910 secure :-1 s=READY pgs=5660 cs=0 l=1 rev1=1 crypto rx=0x7f02a8000c10 tx=0x7f02a8009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a80c6070 con 0x7f02b81d1200 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 10 monclient: dump: 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:31:10.442 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f02a80c2290 con 0x7f02b81d1200 2024-09-10T20:31:10.443 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d1200 msgr2=0x7f02b809e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d1200 0x7f02b809e910 secure :-1 s=READY pgs=5660 cs=0 l=1 rev1=1 crypto rx=0x7f02a8000c10 tx=0x7f02a8009240 comp rx=0 tx=0).stop 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d1200 msgr2=0x7f02b809e910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02b57fa640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f02b81d1200 0x7f02b809e910 secure :-1 s=CLOSED pgs=5660 cs=0 l=1 rev1=1 crypto rx=0x7f02a8000c10 tx=0x7f02a8009240 comp rx=0 tx=0).stop 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:31:10.444 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 28, 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 317078, 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:31:10.445 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:31:10.446 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:31:10.447 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:31:10.448 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.449 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:31:10.450 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:31:10.451 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.452 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:31:10.453 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:31:10.454 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.455 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 7 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:31:10.456 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:31:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:31:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:31:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:31:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:31:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:31:10.457 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:31:10.457 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:31:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 10 librados: watch_flush enter 2024-09-10T20:31:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 10 librados: watch_flush exit 2024-09-10T20:31:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 20 client.34179.objecter shutdown clearing up homeless session... 2024-09-10T20:31:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 10 client.34179.objecter successfully canceled tick 2024-09-10T20:31:10.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 >> v2:172.21.15.176:6812/3003473398 conn(0x7f02a403cea0 msgr2=0x7f02a403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 1 --2- 172.21.15.153:0/2103119620 >> v2:172.21.15.176:6812/3003473398 conn(0x7f02a403cea0 0x7f02a403f360 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7f02b8125510 tx=0x7f02b00071c0 comp rx=0 tx=0).stop 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 10 monclient: shutdown 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a4580 msgr2=0x7f02b8123c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 1 --2- 172.21.15.153:0/2103119620 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f02b80a4580 0x7f02b8123c90 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f02ac00c450 tx=0x7f02ac00c910 comp rx=0 tx=0).stop 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bd286640 1 -- 172.21.15.153:0/2103119620 reap_dead start 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bd286640 1 -- 172.21.15.153:0/2103119620 reap_dead start 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 shutdown_connections 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.435+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 >> 172.21.15.153:0/2103119620 conn(0x7f02b80ab7e0 msgr2=0x7f02b80a98a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.439+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 shutdown_connections 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.439+0000 7f02bece9640 1 -- 172.21.15.153:0/2103119620 wait complete. 2024-09-10T20:31:10.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:31:10.439+0000 7f02bece9640 1 librados: shutdown 2024-09-10T20:31:10.466 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T20:31:10.466 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-10T20:31:25.523 INFO:tasks.workunit.client.0.smithi176.stdout:e 2024-09-10T20:31:25.524 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6259 ms) 2024-09-10T20:31:25.524 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-10T20:31:25.524 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7733 ms) 2024-09-10T20:31:25.524 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-10T20:31:25.524 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (6288 ms) 2024-09-10T20:31:25.524 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-10T20:31:25.525 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7004 ms) 2024-09-10T20:31:25.525 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-10T20:31:25.525 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6968 ms) 2024-09-10T20:31:25.525 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-10T20:31:25.525 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7072 ms) 2024-09-10T20:31:25.525 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-10T20:31:25.525 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6947 ms) 2024-09-10T20:31:25.525 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-10T20:31:25.526 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7259 ms) 2024-09-10T20:31:25.526 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-10T20:31:25.526 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (7039 ms) 2024-09-10T20:31:25.526 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-10T20:31:25.526 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (6720 ms) 2024-09-10T20:31:25.526 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-10T20:31:25.526 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (6990 ms) 2024-09-10T20:31:25.526 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-10T20:31:25.527 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7267 ms) 2024-09-10T20:31:25.527 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-10T20:31:25.527 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (7036 ms) 2024-09-10T20:31:25.527 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-10T20:31:25.527 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (25917 ms) 2024-09-10T20:31:25.527 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-10T20:31:25.527 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7010 ms) 2024-09-10T20:31:25.527 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-10T20:31:25.527 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6837 ms) 2024-09-10T20:31:25.528 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (130346 ms total) 2024-09-10T20:31:25.528 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: 2024-09-10T20:31:25.528 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [----------] Global test environment tear-down 2024-09-10T20:31:25.528 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (249465 ms total) 2024-09-10T20:31:25.528 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-10T20:32:15.534 INFO:tasks.workunit.client.0.smithi176.stdout:o.RoundTripAppendPP (3024 ms) 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3021 ms) 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3010 ms) 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (2969 ms) 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: setting pool EIO 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: max_success 128, min_failed 129 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (4053 ms) 2024-09-10T20:32:15.535 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3029 ms) 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (151461 ms total) 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2908 ms) 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (8998 ms) 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15299 ms) 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3016 ms) 2024-09-10T20:32:15.536 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30222 ms total) 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (2699 ms) 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (2699 ms total) 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14020 ms) 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (7278 ms) 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6155 ms) 2024-09-10T20:32:15.537 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (26780 ms) 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3010 ms) 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (6825 ms) 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6986 ms) 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (8126 ms) 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (6903 ms) 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-10T20:32:15.538 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (8126 ms) 2024-09-10T20:32:15.539 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-10T20:32:15.539 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6851 ms) 2024-09-10T20:32:15.539 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-10T20:32:15.539 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7070 ms) 2024-09-10T20:32:15.539 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-10T20:32:15.539 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6965 ms) 2024-09-10T20:32:40.467 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T20:32:40.468 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T20:32:40.468 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T20:32:40.472 INFO:tasks.ceph.mon.c:Started 2024-09-10T20:32:40.473 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-10T20:32:40.473 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-10T20:32:40.473 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-10T20:32:55.474 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-10T20:32:55.474 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-10T20:32:55.474 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T20:32:55.475 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:32:55.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.598+0000 7f8547e8a640 10 monclient: get_monmap_and_config 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.598+0000 7f8547e8a640 10 monclient: build_initial_monmap 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.598+0000 7f8547e8a640 10 monclient: monmap: 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:55.604039+0000 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:55.604039+0000 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:55.608 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:55.609 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:55.609 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:55.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.598+0000 7f8547e8a640 1 Processor -- start 2024-09-10T20:32:55.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.598+0000 7f8547e8a640 1 -- start start 2024-09-10T20:32:55.610 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.598+0000 7f8547e8a640 10 monclient: init 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 10 monclient: _add_conns ranks=[1,2,0,4,3] 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f85400a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a4580 0x7f854013d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 10 monclient(hunting): picked mon.noname-b con 0x7f85400a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f85400a49a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85400a6a20 con 0x7f85400a49a0 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85400a85a0 con 0x7f85400a41a0 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f85400a2570 con 0x7f85400a4580 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8547e8a640 10 monclient(hunting): authenticate will time out at 1335.744379s 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8546427640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f85400a49a0 0x7f854013fd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37302/0 (socket says 172.21.15.153:37302) 2024-09-10T20:32:55.611 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a4580 0x7f854013d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8546427640 1 -- 172.21.15.153:0/1794235319 learned_addr learned my addr 172.21.15.153:0/1794235319 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient(hunting): get_auth_request con 0x7f85400a4580 auth_method 0 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient(hunting): handle_auth_done global_id 44099 payload 274 2024-09-10T20:32:55.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 1 -- 172.21.15.153:0/1794235319 >> v1:172.21.15.176:6789/0 conn(0x7f85400a41a0 legacy=0x7f85400a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:55.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 1 -- 172.21.15.153:0/1794235319 >> v1:172.21.15.153:6789/0 conn(0x7f85400a49a0 legacy=0x7f854013fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:55.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient: _finish_hunting 0 2024-09-10T20:32:55.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 1 monclient: found mon.noname-b 2024-09-10T20:32:55.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:55.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:32:55.613 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.602+0000 7f8545425640 1 -- 172.21.15.153:0/1794235319 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8540143450 con 0x7f85400a4580 2024-09-10T20:32:55.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 1 -- 172.21.15.153:0/1794235319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a4580 msgr2=0x7f854013d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:55.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 1 --2- 172.21.15.153:0/1794235319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a4580 0x7f854013d7c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f85300097f0 tx=0x7f853001c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2258fb0b762fd8b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:55.619 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 1 -- 172.21.15.153:0/1794235319 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85300b0070 con 0x7f85400a4580 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 10 monclient: dump: 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:55.620 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 10 monclient: _finish_auth 0 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 10 monclient: _check_auth_tickets 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 5 monclient: authenticate success, global_id 44099 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 1 -- 172.21.15.153:0/1794235319 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85300a97f0 con 0x7f85400a4580 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:55.621 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 1 -- 172.21.15.153:0/1794235319 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85300a4e60 con 0x7f85400a4580 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 4 set_mon_vals no callback set 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8544c24640 10 monclient: dump: 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:55.622 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient: shutdown 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 -- 172.21.15.153:0/1794235319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a4580 msgr2=0x7f854013d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 --2- 172.21.15.153:0/1794235319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a4580 0x7f854013d7c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f85300097f0 tx=0x7f853001c3a0 comp rx=0 tx=0).stop 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 -- 172.21.15.153:0/1794235319 shutdown_connections 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 --2- 172.21.15.153:0/1794235319 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a4580 0x7f854013d7c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 -- 172.21.15.153:0/1794235319 >> 172.21.15.153:0/1794235319 conn(0x7f85400ab7e0 msgr2=0x7f85400a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 -- 172.21.15.153:0/1794235319 shutdown_connections 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 -- 172.21.15.153:0/1794235319 wait complete. 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient: build_initial_monmap 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient: monmap: 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:55.623 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 librados: starting msgr at 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 librados: starting objecter 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 Processor -- start 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 -- start start 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 librados: setting wanted keys 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 librados: calling monclient init 2024-09-10T20:32:55.624 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient: init 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85400a41a0 0x7f8540128840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient(hunting): picked mon.noname-a con 0x7f85400a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a49a0 0x7f8540126700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient(hunting): picked mon.noname-c con 0x7f85400a49a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85401406c0 0x7f85401d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient(hunting): picked mon.noname-b con 0x7f85401406c0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8540144040 con 0x7f85400a41a0 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8540070400 con 0x7f85401406c0 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:55.625 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f85401441c0 con 0x7f85400a49a0 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8547e8a640 10 monclient(hunting): authenticate will time out at 1335.751580s 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85400a41a0 0x7f8540128840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a49a0 0x7f8540126700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85400a41a0 0x7f8540128840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55264/0 (socket says 172.21.15.153:55264) 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 1 -- 172.21.15.153:0/226425750 learned_addr learned my addr 172.21.15.153:0/226425750 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a49a0 0x7f8540126700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46258/0 (socket says 172.21.15.153:46258) 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 10 monclient(hunting): get_auth_request con 0x7f85400a41a0 auth_method 0 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 10 monclient(hunting): get_auth_request con 0x7f85400a49a0 auth_method 0 2024-09-10T20:32:55.626 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545c26640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8545425640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.606+0000 7f8546427640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85401406c0 0x7f85401d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 10 monclient(hunting): handle_auth_done global_id 34245 payload 824 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 1 -- 172.21.15.153:0/226425750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a49a0 msgr2=0x7f8540126700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a49a0 0x7f8540126700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 1 -- 172.21.15.153:0/226425750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85401406c0 msgr2=0x7f85401d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:55.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85401406c0 0x7f85401d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 10 monclient: _finish_hunting 0 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 1 monclient: found mon.noname-a 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 1 -- 172.21.15.153:0/226425750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85401291d0 con 0x7f85400a41a0 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8546427640 10 monclient: get_auth_request con 0x7f85401406c0 auth_method 0 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8546427640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85401406c0 0x7f85401d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545425640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a49a0 0x7f8540126700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 1 -- 172.21.15.153:0/226425750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85400a41a0 msgr2=0x7f8540128840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8545c26640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85400a41a0 0x7f8540128840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f853c00c450 tx=0x7f853c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 client.?.objecter ms_handle_connect 0x7f85400a41a0 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:55.628 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 1 -- 172.21.15.153:0/226425750 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c012a40 con 0x7f85400a41a0 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 monclient: dump: 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 monclient: _finish_auth 0 2024-09-10T20:32:55.629 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 monclient: _check_auth_tickets 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 5 monclient: authenticate success, global_id 34245 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 10 monclient: _renew_subs 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 1 -- 172.21.15.153:0/226425750 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853c017070 con 0x7f85400a41a0 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85401276a0 con 0x7f85400a41a0 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 1 -- 172.21.15.153:0/226425750 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c012e40 con 0x7f85400a41a0 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 10 monclient: dump: 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:55.630 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85377fe640 4 set_mon_vals no callback set 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 10 client.34245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 10 monclient: _renew_subs 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f854012a040 con 0x7f85400a41a0 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 1 librados: init done 2024-09-10T20:32:55.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85400a3720 con 0x7f85400a41a0 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 1 -- 172.21.15.153:0/226425750 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f853c01c460 con 0x7f85400a41a0 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.610+0000 7f85367fc640 1 --2- 172.21.15.153:0/226425750 >> v2:172.21.15.176:6812/3003473398 conn(0x7f852403ced0 0x7f852403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.614+0000 7f85367fc640 1 -- 172.21.15.153:0/226425750 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(297..297 src has 1..297) ==== 4694+0+0 (secure 0 0 0) 0x7f853c04f410 con 0x7f85400a41a0 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.614+0000 7f85367fc640 10 client.34245.objecter ms_dispatch 0x7f8540000ba0 osd_map(297..297 src has 1..297) 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.614+0000 7f85367fc640 3 client.34245.objecter handle_osd_map got epochs [297,297] > 0 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.614+0000 7f85367fc640 3 client.34245.objecter handle_osd_map decoding full epoch 297 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.614+0000 7f85367fc640 20 client.34245.objecter dump_active .. 0 homeless 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.614+0000 7f8545425640 1 --2- 172.21.15.153:0/226425750 >> v2:172.21.15.176:6812/3003473398 conn(0x7f852403ced0 0x7f852403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:55.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.614+0000 7f8545425640 10 monclient: get_auth_request con 0x7f852403ced0 auth_method 0 2024-09-10T20:32:55.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.614+0000 7f8545425640 1 --2- 172.21.15.153:0/226425750 >> v2:172.21.15.176:6812/3003473398 conn(0x7f852403ced0 0x7f852403f390 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f85300ab000 tx=0x7f85300a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:55.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.614+0000 7f85367fc640 10 client.34245.objecter ms_handle_connect 0x7f852403ced0 2024-09-10T20:32:55.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.618+0000 7f85367fc640 1 -- 172.21.15.153:0/226425750 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f853c01b3c0 con 0x7f85400a41a0 2024-09-10T20:32:55.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.618+0000 7f85367fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:55.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.618+0000 7f85367fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:55.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.854+0000 7f8547e8a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:32:55.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.854+0000 7f8547e8a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:32:55.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.854+0000 7f8547e8a640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:55.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.854+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f854006cd40 con 0x7f85400a41a0 2024-09-10T20:32:55.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.854+0000 7f85367fc640 1 -- 172.21.15.153:0/226425750 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f853c04f080 con 0x7f85400a41a0 2024-09-10T20:32:55.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.854+0000 7f85367fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:32:55.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.854+0000 7f85367fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:55.863 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:32:55.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 10 librados: watch_flush enter 2024-09-10T20:32:55.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 10 librados: watch_flush exit 2024-09-10T20:32:55.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 20 client.34245.objecter shutdown clearing up homeless session... 2024-09-10T20:32:55.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 10 client.34245.objecter successfully canceled tick 2024-09-10T20:32:55.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 >> v2:172.21.15.176:6812/3003473398 conn(0x7f852403ced0 msgr2=0x7f852403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:55.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 --2- 172.21.15.153:0/226425750 >> v2:172.21.15.176:6812/3003473398 conn(0x7f852403ced0 0x7f852403f390 secure :-1 s=READY pgs=3346 cs=0 l=1 rev1=1 crypto rx=0x7f85300ab000 tx=0x7f85300a6000 comp rx=0 tx=0).stop 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 10 monclient: shutdown 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85400a41a0 msgr2=0x7f8540128840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85400a41a0 0x7f8540128840 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f853c00c450 tx=0x7f853c00c910 comp rx=0 tx=0).stop 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 shutdown_connections 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f85401406c0 0x7f85401d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 --2- 172.21.15.153:0/226425750 >> v2:172.21.15.176:6812/3003473398 conn(0x7f852403ced0 0x7f852403f390 unknown :-1 s=CLOSED pgs=3346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f85400a49a0 0x7f8540126700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 --2- 172.21.15.153:0/226425750 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f85400a41a0 0x7f8540128840 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:55.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 >> 172.21.15.153:0/226425750 conn(0x7f85400ab7e0 msgr2=0x7f85400a11e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 shutdown_connections 2024-09-10T20:32:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 -- 172.21.15.153:0/226425750 wait complete. 2024-09-10T20:32:55.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.858+0000 7f8547e8a640 1 librados: shutdown 2024-09-10T20:32:55.880 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T20:32:55.880 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T20:32:55.881 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:32:55.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient: get_monmap_and_config 2024-09-10T20:32:55.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient: build_initial_monmap 2024-09-10T20:32:55.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient: monmap: 2024-09-10T20:32:55.987 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:55.986629+0000 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:55.986629+0000 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:55.988 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:55.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 1 Processor -- start 2024-09-10T20:32:55.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 1 -- start start 2024-09-10T20:32:55.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient: init 2024-09-10T20:32:55.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:55.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-10T20:32:55.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f07700a41a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:32:55.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f07700a4580 addr v1:172.21.15.176:6789/0 2024-09-10T20:32:55.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f07700a49d0 0x7f0770140ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient(hunting): picked mon.noname-a con 0x7f07700a49d0 addr v2:172.21.15.153:3300/0 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07700a6a20 con 0x7f07700a41a0 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07700a85a0 con 0x7f07700a4580 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f07700a2570 con 0x7f07700a49d0 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f07759ad640 10 monclient(hunting): authenticate will time out at 1336.126765s 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f07700a49d0 0x7f0770140ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076effd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f07700a41a0 0x7f07700a3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37318/0 (socket says 172.21.15.153:37318) 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076effd640 1 -- 172.21.15.153:0/1903889893 learned_addr learned my addr 172.21.15.153:0/1903889893 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient(hunting): get_auth_request con 0x7f07700a49d0 auth_method 0 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:55.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient(hunting): handle_auth_done global_id 34248 payload 274 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 1 -- 172.21.15.153:0/1903889893 >> v1:172.21.15.176:6789/0 conn(0x7f07700a4580 legacy=0x7f077013d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 1 -- 172.21.15.153:0/1903889893 >> v1:172.21.15.153:6789/0 conn(0x7f07700a41a0 legacy=0x7f07700a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 1 monclient: found mon.noname-a 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:55.992 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:55.993 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.982+0000 7f076f7fe640 1 -- 172.21.15.153:0/1903889893 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0770143450 con 0x7f07700a49d0 2024-09-10T20:32:55.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076f7fe640 1 -- 172.21.15.153:0/1903889893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 msgr2=0x7f0770140ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:55.994 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076f7fe640 1 --2- 172.21.15.153:0/1903889893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 0x7f0770140ef0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f07600097c0 tx=0x7f076001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4050e23ffb63e6f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 1 -- 172.21.15.153:0/1903889893 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07600b0070 con 0x7f07700a49d0 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 10 monclient: dump: 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:55.996 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 10 monclient: _finish_auth 0 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 10 monclient: _check_auth_tickets 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 1 -- 172.21.15.153:0/1903889893 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07600a97f0 con 0x7f07700a49d0 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 1 -- 172.21.15.153:0/1903889893 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07600a9bc0 con 0x7f07700a49d0 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f07759ad640 5 monclient: authenticate success, global_id 34248 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f07759ad640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f07759ad640 4 set_mon_vals no callback set 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:55.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.986+0000 7f076dffb640 10 monclient: dump: 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient: shutdown 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 -- 172.21.15.153:0/1903889893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 msgr2=0x7f0770140ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:55.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 --2- 172.21.15.153:0/1903889893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 0x7f0770140ef0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f07600097c0 tx=0x7f076001c360 comp rx=0 tx=0).stop 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 -- 172.21.15.153:0/1903889893 shutdown_connections 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 --2- 172.21.15.153:0/1903889893 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 0x7f0770140ef0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 -- 172.21.15.153:0/1903889893 >> 172.21.15.153:0/1903889893 conn(0x7f07700ab7e0 msgr2=0x7f07700a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 -- 172.21.15.153:0/1903889893 shutdown_connections 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 -- 172.21.15.153:0/1903889893 wait complete. 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient: build_initial_monmap 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient: monmap: 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:55.999 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 librados: starting msgr at 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 librados: starting objecter 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 Processor -- start 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 -- start start 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 librados: setting wanted keys 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 librados: calling monclient init 2024-09-10T20:32:56.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient: init 2024-09-10T20:32:56.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:56.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:32:56.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07700a41a0 0x7f077013f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient(hunting): picked mon.noname-c con 0x7f07700a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:56.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07700a4580 0x7f07701378c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient(hunting): picked mon.noname-b con 0x7f07700a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:56.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 0x7f07701d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient(hunting): picked mon.noname-a con 0x7f07700a49d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:56.005 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0770144040 con 0x7f07700a49d0 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0770070400 con 0x7f07700a4580 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f07701441c0 con 0x7f07700a41a0 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f07759ad640 10 monclient(hunting): authenticate will time out at 1336.134039s 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07700a41a0 0x7f077013f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 0x7f07701d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07700a41a0 0x7f077013f540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46268/0 (socket says 172.21.15.153:46268) 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 1 -- 172.21.15.153:0/3773808173 learned_addr learned my addr 172.21.15.153:0/3773808173 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 10 monclient(hunting): get_auth_request con 0x7f07700a41a0 auth_method 0 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient(hunting): get_auth_request con 0x7f07700a49d0 auth_method 0 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient(hunting): handle_auth_done global_id 34251 payload 824 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076e7fc640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07700a4580 0x7f07701378c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07700a41a0 msgr2=0x7f077013f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07700a41a0 0x7f077013f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07700a4580 msgr2=0x7f07701378c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07700a4580 0x7f07701378c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 1 monclient: found mon.noname-a 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076f7fe640 1 -- 172.21.15.153:0/3773808173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0770140a60 con 0x7f07700a49d0 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076e7fc640 10 monclient: get_auth_request con 0x7f07700a4580 auth_method 0 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076effd640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f07700a41a0 0x7f077013f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.990+0000 7f076e7fc640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f07700a4580 0x7f07701378c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076f7fe640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 msgr2=0x7f07701d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076f7fe640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 0x7f07701d0350 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f07600ab000 tx=0x7f076001aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 client.?.objecter ms_handle_connect 0x7f07700a49d0 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07600b0050 con 0x7f07700a49d0 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 monclient: dump: 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.009 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 monclient: _finish_auth 0 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 monclient: _check_auth_tickets 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07600bc070 con 0x7f07700a49d0 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 5 monclient: authenticate success, global_id 34251 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 10 monclient: _renew_subs 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0770140d40 con 0x7f07700a49d0 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07600b7450 con 0x7f07700a49d0 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 monclient: dump: 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 10 client.34251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 10 monclient: _renew_subs 2024-09-10T20:32:56.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076cff9640 4 set_mon_vals no callback set 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f077013e9e0 con 0x7f07700a49d0 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 1 librados: init done 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077013d3d0 0x7f07701d0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f07700a21f0 con 0x7f077013d3d0 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077013d3d0 0x7f07701d0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 10 monclient: get_auth_request con 0x7f077013d3d0 auth_method 0 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f07600a96b0 con 0x7f07700a49d0 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 10 monclient(hunting): handle_auth_done global_id 34254 payload 824 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076effd640 1 -- 172.21.15.153:0/3773808173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f07700a3720 con 0x7f077013d3d0 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 1 --2- 172.21.15.153:0/3773808173 >> v2:172.21.15.176:6812/3003473398 conn(0x7f075003cea0 0x7f075003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(297..297 src has 1..297) ==== 4694+0+0 (secure 0 0 0) 0x7f07600ea310 con 0x7f07700a49d0 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 10 client.34251.objecter ms_dispatch 0x7f0770000ba0 osd_map(297..297 src has 1..297) 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 3 client.34251.objecter handle_osd_map got epochs [297,297] > 0 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 3 client.34251.objecter handle_osd_map decoding full epoch 297 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f0767fff640 20 client.34251.objecter dump_active .. 0 homeless 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076e7fc640 1 --2- 172.21.15.153:0/3773808173 >> v2:172.21.15.176:6812/3003473398 conn(0x7f075003cea0 0x7f075003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.994+0000 7f076e7fc640 10 monclient: get_auth_request con 0x7f075003cea0 auth_method 0 2024-09-10T20:32:56.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 10 client.34251.objecter ms_handle_connect 0x7f077013d3d0 2024-09-10T20:32:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 10 client.34251.objecter resend_mon_ops 2024-09-10T20:32:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f076effd640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077013d3d0 msgr2=0x7f07701d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f076effd640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077013d3d0 0x7f07701d0890 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f076800d3d0 tx=0x7f076800d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07680075b0 con 0x7f077013d3d0 2024-09-10T20:32:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 10 monclient: dump: 2024-09-10T20:32:56.014 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0768018070 con 0x7f077013d3d0 2024-09-10T20:32:56.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:56.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:56.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077013d3d0 msgr2=0x7f07701d0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077013d3d0 0x7f07701d0890 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f076800d3d0 tx=0x7f076800d890 comp rx=0 tx=0).stop 2024-09-10T20:32:56.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077013d3d0 msgr2=0x7f07701d0890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:56.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f077013d3d0 0x7f07701d0890 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f076e7fc640 1 --2- 172.21.15.153:0/3773808173 >> v2:172.21.15.176:6812/3003473398 conn(0x7f075003cea0 0x7f075003f360 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f0770140f50 tx=0x7f0758009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:55.998+0000 7f0767fff640 10 client.34251.objecter ms_handle_connect 0x7f075003cea0 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f07759ad640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f07759ad640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07701d0dd0 0x7f077009e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f07759ad640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f07701d11e0 con 0x7f07701d0dd0 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07701d0dd0 0x7f077009e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 10 monclient: get_auth_request con 0x7f07701d0dd0 auth_method 0 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 10 monclient(hunting): handle_auth_done global_id 34257 payload 824 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.002+0000 7f076effd640 1 -- 172.21.15.153:0/3773808173 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0770140650 con 0x7f07701d0dd0 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 10 client.34251.objecter ms_handle_connect 0x7f07701d0dd0 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 10 client.34251.objecter resend_mon_ops 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f076effd640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07701d0dd0 msgr2=0x7f077009e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f076effd640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07701d0dd0 0x7f077009e190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f076800d3d0 tx=0x7f0768004cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768013810 con 0x7f07701d0dd0 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 10 monclient: dump: 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.020 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f0768013810 con 0x7f07701d0dd0 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07701d0dd0 msgr2=0x7f077009e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07701d0dd0 0x7f077009e190 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f076800d3d0 tx=0x7f0768004cd0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07701d0dd0 msgr2=0x7f077009e190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:56.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f0767fff640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07701d0dd0 0x7f077009e190 secure :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f076800d3d0 tx=0x7f0768004cd0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.022 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:32:56.022 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:32:56.022 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:32:56.022 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:32:56.022 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 32, 2024-09-10T20:32:56.022 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 136566, 2024-09-10T20:32:56.022 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:56.023 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:56.024 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:32:56.025 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:56.026 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:32:56.027 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.028 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.029 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.030 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.031 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.032 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.033 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:56.034 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:56.034 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:32:56.034 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.034 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.034 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.034 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:32:56.034 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:32:56.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 10 librados: watch_flush enter 2024-09-10T20:32:56.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 10 librados: watch_flush exit 2024-09-10T20:32:56.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 20 client.34251.objecter shutdown clearing up homeless session... 2024-09-10T20:32:56.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 10 client.34251.objecter successfully canceled tick 2024-09-10T20:32:56.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 >> v2:172.21.15.176:6812/3003473398 conn(0x7f075003cea0 msgr2=0x7f075003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 1 --2- 172.21.15.153:0/3773808173 >> v2:172.21.15.176:6812/3003473398 conn(0x7f075003cea0 0x7f075003f360 secure :-1 s=READY pgs=3347 cs=0 l=1 rev1=1 crypto rx=0x7f0770140f50 tx=0x7f0758009040 comp rx=0 tx=0).stop 2024-09-10T20:32:56.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 10 monclient: shutdown 2024-09-10T20:32:56.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:56.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 msgr2=0x7f07701d0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 1 --2- 172.21.15.153:0/3773808173 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f07700a49d0 0x7f07701d0350 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f07600ab000 tx=0x7f076001aba0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f076f7fe640 1 -- 172.21.15.153:0/3773808173 reap_dead start 2024-09-10T20:32:56.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f076f7fe640 1 -- 172.21.15.153:0/3773808173 reap_dead start 2024-09-10T20:32:56.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 shutdown_connections 2024-09-10T20:32:56.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 >> 172.21.15.153:0/3773808173 conn(0x7f07700ab7e0 msgr2=0x7f077013aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:56.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 shutdown_connections 2024-09-10T20:32:56.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 1 -- 172.21.15.153:0/3773808173 wait complete. 2024-09-10T20:32:56.037 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.006+0000 7f07759ad640 1 librados: shutdown 2024-09-10T20:32:56.037 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:32:56.037 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T20:32:56.038 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T20:32:56.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.138+0000 7feb522d3640 10 monclient: get_monmap_and_config 2024-09-10T20:32:56.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.138+0000 7feb522d3640 10 monclient: build_initial_monmap 2024-09-10T20:32:56.146 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.138+0000 7feb522d3640 10 monclient: monmap: 2024-09-10T20:32:56.146 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:56.146 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.146 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:56.145425+0000 2024-09-10T20:32:56.146 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:56.145425+0000 2024-09-10T20:32:56.147 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:56.147 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.147 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:56.147 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:56.147 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:56.147 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:56.147 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:56.147 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 1 Processor -- start 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 1 -- start start 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient: init 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient: _add_conns ranks=[4,3,2,1,0] 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7feb4c0a28c0 0x7feb4c09d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient(hunting): picked mon.noname-c con 0x7feb4c0a28c0 addr v2:172.21.15.176:3300/0 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feb4c0a3060 0x7feb4c09d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient(hunting): picked mon.noname-a con 0x7feb4c0a3060 addr v2:172.21.15.153:3300/0 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c09dab0 0x7feb4c135580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.149 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient(hunting): picked mon.noname-b con 0x7feb4c09dab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7feb4c09f640 con 0x7feb4c0a3060 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7feb4c0a34e0 con 0x7feb4c0a28c0 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c0a2ca0 con 0x7feb4c09dab0 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb522d3640 10 monclient(hunting): authenticate will time out at 1336.284867s 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb50870640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c09dab0 0x7feb4c135580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feb4c0a3060 0x7feb4c09d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.150 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb50870640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c09dab0 msgr2=0x7feb4c135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb50870640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c09dab0 msgr2=0x7feb4c135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb50870640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c09dab0 0x7feb4c135580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7feb4c0a3060 0x7feb4c09d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55324/0 (socket says 172.21.15.153:55324) 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 1 -- 172.21.15.153:0/2457434222 learned_addr learned my addr 172.21.15.153:0/2457434222 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient(hunting): get_auth_request con 0x7feb4c0a3060 auth_method 0 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb50870640 1 --2- 172.21.15.153:0/2457434222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c09dab0 0x7feb4c135580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.151 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb4bfff640 1 --2- 172.21.15.153:0/2457434222 >> v2:172.21.15.176:3300/0 conn(0x7feb4c0a28c0 0x7feb4c09d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient(hunting): handle_auth_done global_id 34260 payload 274 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 1 -- 172.21.15.153:0/2457434222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c09dab0 msgr2=0x7feb4c135580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 1 --2- 172.21.15.153:0/2457434222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c09dab0 0x7feb4c135580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 1 -- 172.21.15.153:0/2457434222 >> v2:172.21.15.176:3300/0 conn(0x7feb4c0a28c0 msgr2=0x7feb4c09d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 1 --2- 172.21.15.153:0/2457434222 >> v2:172.21.15.176:3300/0 conn(0x7feb4c0a28c0 0x7feb4c09d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient: _finish_hunting 0 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 1 monclient: found mon.noname-a 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb437fe640 1 -- 172.21.15.153:0/2457434222 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c09f890 con 0x7feb4c0a3060 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb4bfff640 10 monclient: get_auth_request con 0x7feb4c0a28c0 auth_method 0 2024-09-10T20:32:56.152 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.142+0000 7feb4bfff640 1 --2- 172.21.15.153:0/2457434222 >> v2:172.21.15.176:3300/0 conn(0x7feb4c0a28c0 0x7feb4c09d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 1 -- 172.21.15.153:0/2457434222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a3060 msgr2=0x7feb4c09d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 1 --2- 172.21.15.153:0/2457434222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a3060 0x7feb4c09d570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feb4c06bfc0 tx=0x7feb3401e7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5392ed4d57e89d3a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 1 -- 172.21.15.153:0/2457434222 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb340b4070 con 0x7feb4c0a3060 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 10 monclient: dump: 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.156 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 1 -- 172.21.15.153:0/2457434222 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb340a7960 con 0x7feb4c0a3060 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 5 monclient: authenticate success, global_id 34260 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 1 -- 172.21.15.153:0/2457434222 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb340a7d30 con 0x7feb4c0a3060 2024-09-10T20:32:56.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 4 set_mon_vals no callback set 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb4b7fe640 10 monclient: dump: 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient: shutdown 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:56.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 -- 172.21.15.153:0/2457434222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a3060 msgr2=0x7feb4c09d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 --2- 172.21.15.153:0/2457434222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a3060 0x7feb4c09d570 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7feb4c06bfc0 tx=0x7feb3401e7f0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 -- 172.21.15.153:0/2457434222 shutdown_connections 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 --2- 172.21.15.153:0/2457434222 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c09dab0 0x7feb4c135580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 --2- 172.21.15.153:0/2457434222 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a3060 0x7feb4c09d570 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 --2- 172.21.15.153:0/2457434222 >> v2:172.21.15.176:3300/0 conn(0x7feb4c0a28c0 0x7feb4c09d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 -- 172.21.15.153:0/2457434222 >> 172.21.15.153:0/2457434222 conn(0x7feb4c0a71b0 msgr2=0x7feb4c0a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 -- 172.21.15.153:0/2457434222 shutdown_connections 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 -- 172.21.15.153:0/2457434222 wait complete. 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient: build_initial_monmap 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient: monmap: 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:56.160 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 librados: starting msgr at 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 librados: starting objecter 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 Processor -- start 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 -- start start 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 librados: setting wanted keys 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 librados: calling monclient init 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient: init 2024-09-10T20:32:56.161 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb4c09dab0 0x7feb4c0721f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient(hunting): picked mon.noname-b con 0x7feb4c09dab0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a28c0 0x7feb4c072730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient(hunting): picked mon.noname-a con 0x7feb4c0a28c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c0a3060 0x7feb4c072c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient(hunting): picked mon.noname-c con 0x7feb4c0a3060 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c0a1b40 con 0x7feb4c0a28c0 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7feb4c0a19c0 con 0x7feb4c09dab0 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c0a1cc0 con 0x7feb4c0a3060 2024-09-10T20:32:56.162 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb522d3640 10 monclient(hunting): authenticate will time out at 1336.291814s 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a28c0 0x7feb4c072730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a28c0 0x7feb4c072730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55330/0 (socket says 172.21.15.153:55330) 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 1 -- 172.21.15.153:0/1141081744 learned_addr learned my addr 172.21.15.153:0/1141081744 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 10 monclient(hunting): get_auth_request con 0x7feb4c0a28c0 auth_method 0 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb50870640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c0a3060 0x7feb4c072c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb50870640 10 monclient(hunting): get_auth_request con 0x7feb4c0a3060 auth_method 0 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb50870640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb50870640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.163 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb50870640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.146+0000 7feb437fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb50870640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb50870640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb50870640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb4c09dab0 0x7feb4c0721f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 10 monclient(hunting): handle_auth_done global_id 34263 payload 824 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c0a3060 msgr2=0x7feb4c072c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c0a3060 0x7feb4c072c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb4c09dab0 msgr2=0x7feb4c0721f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb4c09dab0 0x7feb4c0721f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 10 monclient: _finish_hunting 0 2024-09-10T20:32:56.164 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 1 monclient: found mon.noname-a 2024-09-10T20:32:56.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:56.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 1 -- 172.21.15.153:0/1141081744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb4c11db30 con 0x7feb4c0a28c0 2024-09-10T20:32:56.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb50870640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c0a3060 0x7feb4c072c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:32:56.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 10 monclient: get_auth_request con 0x7feb4c09dab0 auth_method 0 2024-09-10T20:32:56.165 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7feb4c09dab0 0x7feb4c0721f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:32:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a28c0 msgr2=0x7feb4c072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb437fe640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a28c0 0x7feb4c072730 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb3401cea0 tx=0x7feb3401d030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 client.?.objecter ms_handle_connect 0x7feb4c0a28c0 2024-09-10T20:32:56.169 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb340bf070 con 0x7feb4c0a28c0 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 monclient: dump: 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.170 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 monclient: _finish_auth 0 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 monclient: _check_auth_tickets 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb340afe30 con 0x7feb4c0a28c0 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb340b4070 con 0x7feb4c0a28c0 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 monclient: dump: 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.171 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 5 monclient: authenticate success, global_id 34263 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 10 monclient: _renew_subs 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4affd640 4 set_mon_vals no callback set 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb4c12f3f0 con 0x7feb4c0a28c0 2024-09-10T20:32:56.172 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 10 client.34263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 10 monclient: _renew_subs 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb4c13b590 con 0x7feb4c0a28c0 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 1 librados: init done 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7feb340b93e0 con 0x7feb4c0a28c0 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12dea0 0x7feb4c12e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c135580 con 0x7feb4c12dea0 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12dea0 0x7feb4c12e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 10 monclient: get_auth_request con 0x7feb4c12dea0 auth_method 0 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.173 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 1 --2- 172.21.15.153:0/1141081744 >> v2:172.21.15.176:6812/3003473398 conn(0x7feb2c03cea0 0x7feb2c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(297..297 src has 1..297) ==== 4694+0+0 (secure 0 0 0) 0x7feb340e9bb0 con 0x7feb4c0a28c0 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 10 client.34263.objecter ms_dispatch 0x7feb4c000ba0 osd_map(297..297 src has 1..297) 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 3 client.34263.objecter handle_osd_map got epochs [297,297] > 0 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 3 client.34263.objecter handle_osd_map decoding full epoch 297 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 10 monclient(hunting): handle_auth_done global_id 44111 payload 824 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb497fa640 20 client.34263.objecter dump_active .. 0 homeless 2024-09-10T20:32:56.174 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.150+0000 7feb4bfff640 1 -- 172.21.15.153:0/1141081744 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feb4c073b20 con 0x7feb4c12dea0 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb50870640 1 --2- 172.21.15.153:0/1141081744 >> v2:172.21.15.176:6812/3003473398 conn(0x7feb2c03cea0 0x7feb2c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb50870640 10 monclient: get_auth_request con 0x7feb2c03cea0 auth_method 0 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 10 client.34263.objecter ms_handle_connect 0x7feb4c12dea0 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 10 client.34263.objecter resend_mon_ops 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb4bfff640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12dea0 msgr2=0x7feb4c12e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb4bfff640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12dea0 0x7feb4c12e280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00d790 tx=0x7feb3c00dc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c016070 con 0x7feb4c12dea0 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 10 monclient: dump: 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.175 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb3c008d20 con 0x7feb4c12dea0 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12dea0 msgr2=0x7feb4c12e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12dea0 0x7feb4c12e280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00d790 tx=0x7feb3c00dc50 comp rx=0 tx=0).stop 2024-09-10T20:32:56.176 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12dea0 msgr2=0x7feb4c12e280 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12dea0 0x7feb4c12e280 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00d790 tx=0x7feb3c00dc50 comp rx=0 tx=0).stop 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb497fa640 10 client.34263.objecter ms_handle_connect 0x7feb2c03cea0 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.154+0000 7feb50870640 1 --2- 172.21.15.153:0/1141081744 >> v2:172.21.15.176:6812/3003473398 conn(0x7feb2c03cea0 0x7feb2c03f360 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7feb44008fd0 tx=0x7feb44008660 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb522d3640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb522d3640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12e7c0 0x7feb4c132d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb522d3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7feb4c06a640 con 0x7feb4c12e7c0 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12e7c0 0x7feb4c132d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 10 monclient: get_auth_request con 0x7feb4c12e7c0 auth_method 0 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.177 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 10 monclient(hunting): handle_auth_done global_id 44114 payload 824 2024-09-10T20:32:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 1 -- 172.21.15.153:0/1141081744 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feb4c12eba0 con 0x7feb4c12e7c0 2024-09-10T20:32:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb497fa640 10 client.34263.objecter ms_handle_connect 0x7feb4c12e7c0 2024-09-10T20:32:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb497fa640 10 client.34263.objecter resend_mon_ops 2024-09-10T20:32:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12e7c0 msgr2=0x7feb4c132d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.162+0000 7feb4bfff640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12e7c0 0x7feb4c132d90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00d790 tx=0x7feb3c004740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb3c00de60 con 0x7feb4c12e7c0 2024-09-10T20:32:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 10 monclient: dump: 2024-09-10T20:32:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.179 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7feb3c0113c0 con 0x7feb4c12e7c0 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12e7c0 msgr2=0x7feb4c132d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12e7c0 0x7feb4c132d90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00d790 tx=0x7feb3c004740 comp rx=0 tx=0).stop 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12e7c0 msgr2=0x7feb4c132d90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:56.180 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb497fa640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7feb4c12e7c0 0x7feb4c132d90 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7feb3c00d790 tx=0x7feb3c004740 comp rx=0 tx=0).stop 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 32, 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 15619, 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:32:56.181 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:56.182 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:56.183 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:56.184 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:56.185 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.186 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.187 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:56.188 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:56.189 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.190 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.191 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.192 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.193 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.193 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:32:56.193 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:32:56.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 10 librados: watch_flush enter 2024-09-10T20:32:56.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 10 librados: watch_flush exit 2024-09-10T20:32:56.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 20 client.34263.objecter shutdown clearing up homeless session... 2024-09-10T20:32:56.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 10 client.34263.objecter successfully canceled tick 2024-09-10T20:32:56.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 >> v2:172.21.15.176:6812/3003473398 conn(0x7feb2c03cea0 msgr2=0x7feb2c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 1 --2- 172.21.15.153:0/1141081744 >> v2:172.21.15.176:6812/3003473398 conn(0x7feb2c03cea0 0x7feb2c03f360 secure :-1 s=READY pgs=3348 cs=0 l=1 rev1=1 crypto rx=0x7feb44008fd0 tx=0x7feb44008660 comp rx=0 tx=0).stop 2024-09-10T20:32:56.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 10 monclient: shutdown 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a28c0 msgr2=0x7feb4c072730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 1 --2- 172.21.15.153:0/1141081744 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7feb4c0a28c0 0x7feb4c072730 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7feb3401cea0 tx=0x7feb3401d030 comp rx=0 tx=0).stop 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb50870640 1 -- 172.21.15.153:0/1141081744 reap_dead start 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb50870640 1 -- 172.21.15.153:0/1141081744 reap_dead start 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 shutdown_connections 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 >> 172.21.15.153:0/1141081744 conn(0x7feb4c0a71b0 msgr2=0x7feb4c0a4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 shutdown_connections 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 1 -- 172.21.15.153:0/1141081744 wait complete. 2024-09-10T20:32:56.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.166+0000 7feb522d3640 1 librados: shutdown 2024-09-10T20:32:56.196 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:32:56.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T20:32:56.196 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:32:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.290+0000 7f43f5e4e640 10 monclient: get_monmap_and_config 2024-09-10T20:32:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.290+0000 7f43f5e4e640 10 monclient: build_initial_monmap 2024-09-10T20:32:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.290+0000 7f43f5e4e640 10 monclient: monmap: 2024-09-10T20:32:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:56.297575+0000 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:56.297575+0000 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:56.299 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 1 Processor -- start 2024-09-10T20:32:56.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 1 -- start start 2024-09-10T20:32:56.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient: init 2024-09-10T20:32:56.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient: _add_conns ranks=[1,2,4,3,0] 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f43f00a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a4580 0x7f43f013d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f43f00a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f43f00a49a0 0x7f43f013fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f43f00a49a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43f00a6a20 con 0x7f43f00a41a0 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f43f00a2570 con 0x7f43f00a49a0 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f43f0070400 con 0x7f43f00a4580 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43f5e4e640 10 monclient(hunting): authenticate will time out at 1336.437142s 2024-09-10T20:32:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a4580 0x7f43f013d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a4580 0x7f43f013d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46328/0 (socket says 172.21.15.153:46328) 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 1 -- 172.21.15.153:0/3691168232 learned_addr learned my addr 172.21.15.153:0/3691168232 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient(hunting): get_auth_request con 0x7f43f00a4580 auth_method 0 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43effff640 1 --2- 172.21.15.153:0/3691168232 >> v2:172.21.15.176:3300/0 conn(0x7f43f00a49a0 0x7f43f013fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43effff640 10 monclient(hunting): get_auth_request con 0x7f43f00a49a0 auth_method 0 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43effff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient(hunting): handle_auth_done global_id 44117 payload 274 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 1 -- 172.21.15.153:0/3691168232 >> v2:172.21.15.176:3300/0 conn(0x7f43f00a49a0 msgr2=0x7f43f013fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 1 --2- 172.21.15.153:0/3691168232 >> v2:172.21.15.176:3300/0 conn(0x7f43f00a49a0 0x7f43f013fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 1 -- 172.21.15.153:0/3691168232 >> v1:172.21.15.176:6789/0 conn(0x7f43f00a41a0 legacy=0x7f43f00a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient: _finish_hunting 0 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 1 monclient: found mon.noname-b 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:32:56.304 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.294+0000 7f43eeffd640 1 -- 172.21.15.153:0/3691168232 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f00a85a0 con 0x7f43f00a4580 2024-09-10T20:32:56.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43eeffd640 1 -- 172.21.15.153:0/3691168232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a4580 msgr2=0x7f43f013d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43eeffd640 1 --2- 172.21.15.153:0/3691168232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a4580 0x7f43f013d7c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009870 tx=0x7f43d801c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=affadbc9bcfe6672 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 1 -- 172.21.15.153:0/3691168232 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d80b0070 con 0x7f43f00a4580 2024-09-10T20:32:56.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 10 monclient: dump: 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 10 monclient: _finish_auth 0 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 5 monclient: authenticate success, global_id 44117 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 1 -- 172.21.15.153:0/3691168232 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43d80a9710 con 0x7f43f00a4580 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 1 -- 172.21.15.153:0/3691168232 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43d80a9ae0 con 0x7f43f00a4580 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 4 set_mon_vals no callback set 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43ee7fc640 10 monclient: dump: 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 10 monclient: shutdown 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3691168232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a4580 msgr2=0x7f43f013d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 --2- 172.21.15.153:0/3691168232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a4580 0x7f43f013d7c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f43d8009870 tx=0x7f43d801c420 comp rx=0 tx=0).stop 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3691168232 shutdown_connections 2024-09-10T20:32:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 --2- 172.21.15.153:0/3691168232 >> v2:172.21.15.176:3300/0 conn(0x7f43f00a49a0 0x7f43f013fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 --2- 172.21.15.153:0/3691168232 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a4580 0x7f43f013d7c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3691168232 >> 172.21.15.153:0/3691168232 conn(0x7f43f00ab7e0 msgr2=0x7f43f00a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3691168232 shutdown_connections 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3691168232 wait complete. 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 10 monclient: build_initial_monmap 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 10 monclient: monmap: 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:56.310 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:56.311 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 librados: starting msgr at 2024-09-10T20:32:56.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 librados: starting objecter 2024-09-10T20:32:56.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.298+0000 7f43f5e4e640 1 Processor -- start 2024-09-10T20:32:56.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 -- start start 2024-09-10T20:32:56.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 librados: setting wanted keys 2024-09-10T20:32:56.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 librados: calling monclient init 2024-09-10T20:32:56.311 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient: init 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a41a0 0x7f43f01d0150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient(hunting): picked mon.noname-c con 0x7f43f00a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f00a4580 0x7f43f0138890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient(hunting): picked mon.noname-b con 0x7f43f00a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00a49a0 0x7f43f009e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient(hunting): picked mon.noname-a con 0x7f43f00a49a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.316 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f43f0142e00 con 0x7f43f00a49a0 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f43f0142c80 con 0x7f43f00a4580 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f43f0142f80 con 0x7f43f00a41a0 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient(hunting): authenticate will time out at 1336.444540s 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43ef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a41a0 0x7f43f01d0150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00a49a0 0x7f43f009e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43ef7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a41a0 msgr2=0x7f43f01d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43ef7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a41a0 msgr2=0x7f43f01d0150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:32:56.317 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43ef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a41a0 0x7f43f01d0150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00a49a0 0x7f43f009e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55342/0 (socket says 172.21.15.153:55342) 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43ef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a41a0 0x7f43f01d0150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 -- 172.21.15.153:0/3565634468 learned_addr learned my addr 172.21.15.153:0/3565634468 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient(hunting): get_auth_request con 0x7f43f00a49a0 auth_method 0 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43eeffd640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f00a4580 0x7f43f0138890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43eeffd640 10 monclient(hunting): get_auth_request con 0x7f43f00a4580 auth_method 0 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43eeffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient(hunting): handle_auth_done global_id 34266 payload 824 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a41a0 msgr2=0x7f43f01d0150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f43f00a41a0 0x7f43f01d0150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f00a4580 msgr2=0x7f43f0138890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f00a4580 0x7f43f0138890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient: _finish_hunting 0 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 monclient: found mon.noname-a 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 -- 172.21.15.153:0/3565634468 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43f0122980 con 0x7f43f00a49a0 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00a49a0 msgr2=0x7f43f009e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43effff640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00a49a0 0x7f43f009e970 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f43e000efc0 tx=0x7f43e000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 client.?.objecter ms_handle_connect 0x7f43f00a49a0 2024-09-10T20:32:56.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0018020 con 0x7f43f00a49a0 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 monclient: dump: 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.320 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 monclient: _finish_auth 0 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 monclient: _check_auth_tickets 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43e0014680 con 0x7f43f00a49a0 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 5 monclient: authenticate success, global_id 34266 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient: _renew_subs 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43f0122c60 con 0x7f43f00a49a0 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e0014a70 con 0x7f43f00a49a0 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:56.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43e7fff640 10 monclient: dump: 2024-09-10T20:32:56.322 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.322 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.322 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.322 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.322 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.322 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.322 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.322 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43ed7fa640 4 set_mon_vals no callback set 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 client.34266.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient: _renew_subs 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43f0121620 con 0x7f43f00a49a0 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.302+0000 7f43f5e4e640 1 librados: init done 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43f5e4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43f5e4e640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f0138dd0 0x7f43f009eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43f5e4e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f43e001b020 con 0x7f43f00a49a0 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f43f0123160 con 0x7f43f0138dd0 2024-09-10T20:32:56.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 1 --2- 172.21.15.153:0/3565634468 >> v2:172.21.15.176:6812/3003473398 conn(0x7f43d003cea0 0x7f43d003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(297..297 src has 1..297) ==== 4694+0+0 (secure 0 0 0) 0x7f43e0050400 con 0x7f43f00a49a0 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 10 client.34266.objecter ms_dispatch 0x7f43f0000ba0 osd_map(297..297 src has 1..297) 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 3 client.34266.objecter handle_osd_map got epochs [297,297] > 0 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 3 client.34266.objecter handle_osd_map decoding full epoch 297 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f0138dd0 0x7f43f009eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 20 client.34266.objecter dump_active .. 0 homeless 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 10 monclient: get_auth_request con 0x7f43f0138dd0 auth_method 0 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43eeffd640 1 --2- 172.21.15.153:0/3565634468 >> v2:172.21.15.176:6812/3003473398 conn(0x7f43d003cea0 0x7f43d003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43eeffd640 10 monclient: get_auth_request con 0x7f43d003cea0 auth_method 0 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 10 monclient(hunting): handle_auth_done global_id 20128 payload 824 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 1 -- 172.21.15.153:0/3565634468 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f43f00a3740 con 0x7f43f0138dd0 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 10 client.34266.objecter ms_handle_connect 0x7f43d003cea0 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43eeffd640 1 --2- 172.21.15.153:0/3565634468 >> v2:172.21.15.176:6812/3003473398 conn(0x7f43d003cea0 0x7f43d003f360 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f43f00a8a30 tx=0x7f43d8008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 10 client.34266.objecter ms_handle_connect 0x7f43f0138dd0 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 10 client.34266.objecter resend_mon_ops 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f0138dd0 msgr2=0x7f43f009eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43ef7fe640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f0138dd0 0x7f43f009eeb0 secure :-1 s=READY pgs=5715 cs=0 l=1 rev1=1 crypto rx=0x7f43e800d360 tx=0x7f43e800d820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e8007650 con 0x7f43f0138dd0 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.306+0000 7f43e7fff640 10 monclient: dump: 2024-09-10T20:32:56.325 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.310+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f43e8017070 con 0x7f43f0138dd0 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.310+0000 7f43e7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.310+0000 7f43e7fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.310+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f0138dd0 msgr2=0x7f43f009eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.310+0000 7f43e7fff640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f0138dd0 0x7f43f009eeb0 secure :-1 s=READY pgs=5715 cs=0 l=1 rev1=1 crypto rx=0x7f43e800d360 tx=0x7f43e800d820 comp rx=0 tx=0).stop 2024-09-10T20:32:56.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.310+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f0138dd0 msgr2=0x7f43f009eeb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:56.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.310+0000 7f43e7fff640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f0138dd0 0x7f43f009eeb0 secure :-1 s=CLOSED pgs=5715 cs=0 l=1 rev1=1 crypto rx=0x7f43e800d360 tx=0x7f43e800d820 comp rx=0 tx=0).stop 2024-09-10T20:32:56.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43f5e4e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:32:56.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43f5e4e640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f009f3f0 0x7f43f009f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43f5e4e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f43f006cd40 con 0x7f43f009f3f0 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f009f3f0 0x7f43f009f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 10 monclient: get_auth_request con 0x7f43f009f3f0 auth_method 0 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 10 monclient(hunting): handle_auth_done global_id 20131 payload 824 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.314+0000 7f43ef7fe640 1 -- 172.21.15.153:0/3565634468 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f43f009fd10 con 0x7f43f009f3f0 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43ef7fe640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f009f3f0 msgr2=0x7f43f009f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43ef7fe640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f009f3f0 0x7f43f009f7d0 secure :-1 s=READY pgs=5716 cs=0 l=1 rev1=1 crypto rx=0x7f43e800d360 tx=0x7f43e8003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 10 client.34266.objecter ms_handle_connect 0x7f43f009f3f0 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 10 client.34266.objecter resend_mon_ops 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43e801ce60 con 0x7f43f009f3f0 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 10 monclient: dump: 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.330 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f43e8025410 con 0x7f43f009f3f0 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f009f3f0 msgr2=0x7f43f009f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f009f3f0 0x7f43f009f7d0 secure :-1 s=READY pgs=5716 cs=0 l=1 rev1=1 crypto rx=0x7f43e800d360 tx=0x7f43e8003770 comp rx=0 tx=0).stop 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f009f3f0 msgr2=0x7f43f009f7d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43e7fff640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f43f009f3f0 0x7f43f009f7d0 secure :-1 s=CLOSED pgs=5716 cs=0 l=1 rev1=1 crypto rx=0x7f43e800d360 tx=0x7f43e8003770 comp rx=0 tx=0).stop 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:32:56.331 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 32, 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 422965, 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:32:56.332 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:32:56.333 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.334 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:32:56.335 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:56.336 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.337 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:32:56.338 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.339 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:32:56.340 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:56.341 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:56.342 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:56.343 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:56.344 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:32:56.344 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:32:56.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 10 librados: watch_flush enter 2024-09-10T20:32:56.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 10 librados: watch_flush exit 2024-09-10T20:32:56.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 20 client.34266.objecter shutdown clearing up homeless session... 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 10 client.34266.objecter successfully canceled tick 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 >> v2:172.21.15.176:6812/3003473398 conn(0x7f43d003cea0 msgr2=0x7f43d003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 --2- 172.21.15.153:0/3565634468 >> v2:172.21.15.176:6812/3003473398 conn(0x7f43d003cea0 0x7f43d003f360 secure :-1 s=READY pgs=3349 cs=0 l=1 rev1=1 crypto rx=0x7f43f00a8a30 tx=0x7f43d8008040 comp rx=0 tx=0).stop 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 10 monclient: shutdown 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00a49a0 msgr2=0x7f43f009e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43effff640 1 -- 172.21.15.153:0/3565634468 reap_dead start 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00a49a0 0x7f43f009e970 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f43e000efc0 tx=0x7f43e000c490 comp rx=0 tx=0).stop 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 shutdown_connections 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 --2- 172.21.15.153:0/3565634468 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f43f00a49a0 0x7f43f009e970 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 >> 172.21.15.153:0/3565634468 conn(0x7f43f00ab7e0 msgr2=0x7f43f00a10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 shutdown_connections 2024-09-10T20:32:56.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 -- 172.21.15.153:0/3565634468 wait complete. 2024-09-10T20:32:56.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.318+0000 7f43f5e4e640 1 librados: shutdown 2024-09-10T20:32:56.347 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:32:56.347 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T20:32:56.347 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T20:32:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient: get_monmap_and_config 2024-09-10T20:32:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient: build_initial_monmap 2024-09-10T20:32:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient: monmap: 2024-09-10T20:32:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:56.447908+0000 2024-09-10T20:32:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:56.447908+0000 2024-09-10T20:32:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:56.450 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 1 Processor -- start 2024-09-10T20:32:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 1 -- start start 2024-09-10T20:32:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient: init 2024-09-10T20:32:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient: _add_conns ranks=[3,0,1,2,4] 2024-09-10T20:32:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd9b40a3860 0x7fd9b40a3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient(hunting): picked mon.noname-a con 0x7fd9b40a3860 addr v2:172.21.15.153:3300/0 2024-09-10T20:32:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9b4076fd0 addr v1:172.21.15.153:6789/0 2024-09-10T20:32:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd9b409dca0 addr v1:172.21.15.176:6789/0 2024-09-10T20:32:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9b4139460 con 0x7fd9b4076fd0 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9b40a5400 con 0x7fd9b409dca0 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fd9b409cab0 con 0x7fd9b40a3860 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b9b4c640 10 monclient(hunting): authenticate will time out at 1336.587479s 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd9b40a3860 0x7fd9b40a3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd9b40a3860 0x7fd9b40a3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55346/0 (socket says 172.21.15.153:55346) 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b2ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd9b4076fd0 0x7fd9b4138640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37328/0 (socket says 172.21.15.153:37328) 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 1 -- 172.21.15.153:0/3780653303 learned_addr learned my addr 172.21.15.153:0/3780653303 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient(hunting): get_auth_request con 0x7fd9b40a3860 auth_method 0 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient(hunting): handle_auth_done global_id 34269 payload 274 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 1 -- 172.21.15.153:0/3780653303 >> v1:172.21.15.176:6789/0 conn(0x7fd9b409dca0 legacy=0x7fd9b4138d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 1 -- 172.21.15.153:0/3780653303 >> v1:172.21.15.153:6789/0 conn(0x7fd9b4076fd0 legacy=0x7fd9b4138640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient: _finish_hunting 0 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 1 monclient: found mon.noname-a 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.442+0000 7fd9b37fe640 1 -- 172.21.15.153:0/3780653303 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b409cd90 con 0x7fd9b40a3860 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 -- 172.21.15.153:0/3780653303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40a3860 msgr2=0x7fd9b40a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 --2- 172.21.15.153:0/3780653303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40a3860 0x7fd9b40a3c40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009870 tx=0x7fd9a401c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22bb0f97430d6d92 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 1 -- 172.21.15.153:0/3780653303 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40b0070 con 0x7fd9b40a3860 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 10 monclient: dump: 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.458 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 10 monclient: _finish_auth 0 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 10 monclient: _check_auth_tickets 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 1 -- 172.21.15.153:0/3780653303 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9a40ab600 con 0x7fd9b40a3860 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 5 monclient: authenticate success, global_id 34269 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:32:56.459 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 1 -- 172.21.15.153:0/3780653303 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40ab9d0 con 0x7fd9b40a3860 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 4 set_mon_vals no callback set 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b27fc640 10 monclient: dump: 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.460 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient: shutdown 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/3780653303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40a3860 msgr2=0x7fd9b40a3c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 --2- 172.21.15.153:0/3780653303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40a3860 0x7fd9b40a3c40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fd9a4009870 tx=0x7fd9a401c420 comp rx=0 tx=0).stop 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/3780653303 shutdown_connections 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 --2- 172.21.15.153:0/3780653303 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b40a3860 0x7fd9b40a3c40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/3780653303 >> 172.21.15.153:0/3780653303 conn(0x7fd9b40a8510 msgr2=0x7fd9b40a4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:56.461 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/3780653303 shutdown_connections 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/3780653303 wait complete. 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient: build_initial_monmap 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient: monmap: 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.462 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 librados: starting msgr at 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 librados: starting objecter 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 Processor -- start 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 -- start start 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 librados: setting wanted keys 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 librados: calling monclient init 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient: init 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9b4076fd0 0x7fd9b4120450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient(hunting): picked mon.noname-c con 0x7fd9b4076fd0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b409dca0 0x7fd9b41382d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient(hunting): picked mon.noname-a con 0x7fd9b409dca0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9b40a3860 0x7fd9b411e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.463 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient(hunting): picked mon.noname-b con 0x7fd9b40a3860 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b4144870 con 0x7fd9b409dca0 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd9b41446f0 con 0x7fd9b40a3860 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd9b41449f0 con 0x7fd9b4076fd0 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b9b4c640 10 monclient(hunting): authenticate will time out at 1336.594189s 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9b4076fd0 0x7fd9b4120450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b2ffd640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b409dca0 0x7fd9b41382d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9b4076fd0 0x7fd9b4120450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46332/0 (socket says 172.21.15.153:46332) 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 -- 172.21.15.153:0/1834414073 learned_addr learned my addr 172.21.15.153:0/1834414073 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:56.464 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient(hunting): get_auth_request con 0x7fd9b4076fd0 auth_method 0 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b2ffd640 10 monclient(hunting): get_auth_request con 0x7fd9b409dca0 auth_method 0 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b2ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b2ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b2ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b2ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:56.465 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient(hunting): handle_auth_done global_id 44120 payload 824 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 -- 172.21.15.153:0/1834414073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9b40a3860 msgr2=0x7fd9b411e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 --2- 172.21.15.153:0/1834414073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9b40a3860 0x7fd9b411e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 -- 172.21.15.153:0/1834414073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b409dca0 msgr2=0x7fd9b41382d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 --2- 172.21.15.153:0/1834414073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b409dca0 0x7fd9b41382d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient: _finish_hunting 0 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 monclient: found mon.noname-c 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b37fe640 1 -- 172.21.15.153:0/1834414073 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9b4120040 con 0x7fd9b4076fd0 2024-09-10T20:32:56.466 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.450+0000 7fd9b2ffd640 1 --2- 172.21.15.153:0/1834414073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b409dca0 0x7fd9b41382d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:32:56.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b37fe640 1 -- 172.21.15.153:0/1834414073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9b4076fd0 msgr2=0x7fd9b4120450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:56.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b37fe640 1 --2- 172.21.15.153:0/1834414073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9b4076fd0 0x7fd9b4120450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40ae040 tx=0x7fd9a401ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 client.?.objecter ms_handle_connect 0x7fd9b4076fd0 2024-09-10T20:32:56.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 1 -- 172.21.15.153:0/1834414073 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40ba020 con 0x7fd9b4076fd0 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 monclient: dump: 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.469 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 monclient: _finish_auth 0 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 monclient: _check_auth_tickets 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 5 monclient: authenticate success, global_id 44120 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 1 -- 172.21.15.153:0/1834414073 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9a40b6630 con 0x7fd9b4076fd0 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 1 -- 172.21.15.153:0/1834414073 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9a40b0040 con 0x7fd9b4076fd0 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 10 monclient: _renew_subs 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9b4120de0 con 0x7fd9b4076fd0 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 monclient: dump: 2024-09-10T20:32:56.470 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b1ffb640 4 set_mon_vals no callback set 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 10 client.44120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 10 monclient: _renew_subs 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:56.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9b409f0b0 con 0x7fd9b4076fd0 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 1 librados: init done 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b40a3cc0 con 0x7fd9b4076fd0 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 1 -- 172.21.15.153:0/1834414073 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fd9a40bd020 con 0x7fd9b4076fd0 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 1 --2- 172.21.15.153:0/1834414073 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd99403ced0 0x7fd99403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 1 -- 172.21.15.153:0/1834414073 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(297..297 src has 1..297) ==== 4694+0+0 (secure 0 0 0) 0x7fd9a40e9b20 con 0x7fd9b4076fd0 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 10 client.44120.objecter ms_dispatch 0x7fd9b4000ba0 osd_map(297..297 src has 1..297) 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 3 client.44120.objecter handle_osd_map got epochs [297,297] > 0 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 3 client.44120.objecter handle_osd_map decoding full epoch 297 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9abfff640 20 client.44120.objecter dump_active .. 0 homeless 2024-09-10T20:32:56.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b2ffd640 1 --2- 172.21.15.153:0/1834414073 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd99403ced0 0x7fd99403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:56.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.454+0000 7fd9b2ffd640 10 monclient: get_auth_request con 0x7fd99403ced0 auth_method 0 2024-09-10T20:32:56.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.458+0000 7fd9b2ffd640 1 --2- 172.21.15.153:0/1834414073 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd99403ced0 0x7fd99403f390 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4121e60 tx=0x7fd99c007e70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:56.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.458+0000 7fd9abfff640 10 client.44120.objecter ms_handle_connect 0x7fd99403ced0 2024-09-10T20:32:56.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.458+0000 7fd9abfff640 1 -- 172.21.15.153:0/1834414073 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fd9a40c4030 con 0x7fd9b4076fd0 2024-09-10T20:32:56.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.458+0000 7fd9abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:56.473 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.458+0000 7fd9abfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.686+0000 7fd9b9b4c640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T20:32:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.686+0000 7fd9b9b4c640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T20:32:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.686+0000 7fd9b9b4c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:56.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.686+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd9b40a3cc0 con 0x7fd9b4076fd0 2024-09-10T20:32:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9abfff640 1 -- 172.21.15.153:0/1834414073 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd9a4016130 con 0x7fd9b4076fd0 2024-09-10T20:32:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T20:32:56.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9abfff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:56.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=2890353694,config=3541143443,health=92814587,kv=3615289851,logm=1456741834}) 2024-09-10T20:32:56.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=11,config=2,health=12,kv=10,logm=65} crc {auth=615717688,config=3541143443,health=92814587,kv=3915652716,logm=41289208}) 2024-09-10T20:32:56.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=2890353694,config=3541143443,health=92814587,kv=3615289851,logm=1456741834}) 2024-09-10T20:32:56.696 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=11,config=2,health=11,kv=11,logm=65} crc {auth=615717688,config=3541143443,health=2113251397,kv=3615289851,logm=848023039}) 2024-09-10T20:32:56.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 10 librados: watch_flush enter 2024-09-10T20:32:56.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 10 librados: watch_flush exit 2024-09-10T20:32:56.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 20 client.44120.objecter shutdown clearing up homeless session... 2024-09-10T20:32:56.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 10 client.44120.objecter successfully canceled tick 2024-09-10T20:32:56.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd99403ced0 msgr2=0x7fd99403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 --2- 172.21.15.153:0/1834414073 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd99403ced0 0x7fd99403f390 secure :-1 s=READY pgs=3350 cs=0 l=1 rev1=1 crypto rx=0x7fd9b4121e60 tx=0x7fd99c007e70 comp rx=0 tx=0).stop 2024-09-10T20:32:56.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 10 monclient: shutdown 2024-09-10T20:32:56.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:56.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9b4076fd0 msgr2=0x7fd9b4120450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:56.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 --2- 172.21.15.153:0/1834414073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9b4076fd0 0x7fd9b4120450 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd9a40ae040 tx=0x7fd9a401ac60 comp rx=0 tx=0).stop 2024-09-10T20:32:56.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 shutdown_connections 2024-09-10T20:32:56.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 --2- 172.21.15.153:0/1834414073 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd9b40a3860 0x7fd9b411e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 --2- 172.21.15.153:0/1834414073 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd9b409dca0 0x7fd9b41382d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 --2- 172.21.15.153:0/1834414073 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd99403ced0 0x7fd99403f390 unknown :-1 s=CLOSED pgs=3350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 --2- 172.21.15.153:0/1834414073 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd9b4076fd0 0x7fd9b4120450 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:56.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 >> 172.21.15.153:0/1834414073 conn(0x7fd9b40a8510 msgr2=0x7fd9b40a0090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:56.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.699 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=18307008}) 2024-09-10T20:32:56.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 shutdown_connections 2024-09-10T20:32:56.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3981809424}) 2024-09-10T20:32:56.700 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 -- 172.21.15.153:0/1834414073 wait complete. 2024-09-10T20:32:56.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=18307008}) 2024-09-10T20:32:56.701 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1279865591}) 2024-09-10T20:32:56.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:56.690+0000 7fd9b9b4c640 1 librados: shutdown 2024-09-10T20:32:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985772918}) 2024-09-10T20:32:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=468462652}) 2024-09-10T20:32:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2985772918}) 2024-09-10T20:32:56.702 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=818748481}) 2024-09-10T20:32:56.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706689643}) 2024-09-10T20:32:56.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2529545189}) 2024-09-10T20:32:56.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2706689643}) 2024-09-10T20:32:56.703 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.694+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2260861431}) 2024-09-10T20:32:56.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.698+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.698+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575977033}) 2024-09-10T20:32:56.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.698+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2837907141}) 2024-09-10T20:32:56.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.698+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.698+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2575977033}) 2024-09-10T20:32:56.704 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.698+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2376404864}) 2024-09-10T20:32:56.712 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.712 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688830708}) 2024-09-10T20:32:56.712 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3896770463}) 2024-09-10T20:32:56.712 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.712 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3688830708}) 2024-09-10T20:32:56.712 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2010658635}) 2024-09-10T20:32:56.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914689324}) 2024-09-10T20:32:56.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=609122429}) 2024-09-10T20:32:56.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2914689324}) 2024-09-10T20:32:56.714 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.706+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=745924668}) 2024-09-10T20:32:56.715 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T20:32:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126771949}) 2024-09-10T20:32:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2345193676}) 2024-09-10T20:32:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1126771949}) 2024-09-10T20:32:56.716 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2610020033}) 2024-09-10T20:32:56.717 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.717 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409963495}) 2024-09-10T20:32:56.717 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3502166652}) 2024-09-10T20:32:56.717 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.717 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=409963495}) 2024-09-10T20:32:56.718 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.710+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2879564175}) 2024-09-10T20:32:56.719 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.719 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585908523}) 2024-09-10T20:32:56.719 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2379571590}) 2024-09-10T20:32:56.719 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.719 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=585908523}) 2024-09-10T20:32:56.719 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4145264112}) 2024-09-10T20:32:56.721 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.721 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2091345387}) 2024-09-10T20:32:56.721 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=785920499}) 2024-09-10T20:32:56.721 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.721 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2091345387}) 2024-09-10T20:32:56.721 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.714+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=781088685}) 2024-09-10T20:32:56.722 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839487149}) 2024-09-10T20:32:56.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1747378324}) 2024-09-10T20:32:56.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2839487149}) 2024-09-10T20:32:56.723 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4266407194}) 2024-09-10T20:32:56.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303388724}) 2024-09-10T20:32:56.724 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4081531219}) 2024-09-10T20:32:56.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=303388724}) 2024-09-10T20:32:56.725 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1705081068}) 2024-09-10T20:32:56.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582422622}) 2024-09-10T20:32:56.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3370012474}) 2024-09-10T20:32:56.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582422622}) 2024-09-10T20:32:56.726 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.718+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1856946773}) 2024-09-10T20:32:56.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094235664}) 2024-09-10T20:32:56.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=190828627}) 2024-09-10T20:32:56.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094235664}) 2024-09-10T20:32:56.728 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=189859420}) 2024-09-10T20:32:56.729 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.730 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864280582}) 2024-09-10T20:32:56.730 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=990709810}) 2024-09-10T20:32:56.730 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.730 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=864280582}) 2024-09-10T20:32:56.730 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.722+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3022553198}) 2024-09-10T20:32:56.732 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.732 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=586247883}) 2024-09-10T20:32:56.732 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2995982977}) 2024-09-10T20:32:56.732 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.732 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=586247883}) 2024-09-10T20:32:56.732 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2441431748}) 2024-09-10T20:32:56.733 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.733 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244271662}) 2024-09-10T20:32:56.733 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3111344402}) 2024-09-10T20:32:56.733 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.733 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4244271662}) 2024-09-10T20:32:56.733 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.726+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3786001781}) 2024-09-10T20:32:56.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833115113}) 2024-09-10T20:32:56.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=594235822}) 2024-09-10T20:32:56.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=833115113}) 2024-09-10T20:32:56.735 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2269222118}) 2024-09-10T20:32:56.736 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.737 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799573480}) 2024-09-10T20:32:56.737 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3314293874}) 2024-09-10T20:32:56.737 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.737 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2799573480}) 2024-09-10T20:32:56.737 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3620208043}) 2024-09-10T20:32:56.738 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.738 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682907850}) 2024-09-10T20:32:56.738 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2430335565}) 2024-09-10T20:32:56.738 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.739 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3682907850}) 2024-09-10T20:32:56.739 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.730+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3385494003}) 2024-09-10T20:32:56.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241549168}) 2024-09-10T20:32:56.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2011088320}) 2024-09-10T20:32:56.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1241549168}) 2024-09-10T20:32:56.740 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3175236158}) 2024-09-10T20:32:56.742 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.742 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436430976}) 2024-09-10T20:32:56.742 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1419944908}) 2024-09-10T20:32:56.742 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.742 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436430976}) 2024-09-10T20:32:56.742 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.734+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1836775652}) 2024-09-10T20:32:56.744 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.744 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127011708}) 2024-09-10T20:32:56.744 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=34608903}) 2024-09-10T20:32:56.744 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.744 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4127011708}) 2024-09-10T20:32:56.744 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2702378859}) 2024-09-10T20:32:56.745 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.745 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291328263}) 2024-09-10T20:32:56.745 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1696535823}) 2024-09-10T20:32:56.745 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.745 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3291328263}) 2024-09-10T20:32:56.746 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.738+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1318271772}) 2024-09-10T20:32:56.747 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.747 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303351027}) 2024-09-10T20:32:56.747 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2702873727}) 2024-09-10T20:32:56.747 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.747 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2303351027}) 2024-09-10T20:32:56.747 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1406517185}) 2024-09-10T20:32:56.749 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.749 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411717597}) 2024-09-10T20:32:56.749 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3343732222}) 2024-09-10T20:32:56.749 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.749 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1411717597}) 2024-09-10T20:32:56.749 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.742+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2454666306}) 2024-09-10T20:32:56.751 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.751 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=2,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=19} crc {logm=674639863,mdsmap=3140514009,mgr=1914389234,mgr_metadata=3232925821,mgrstat=4158154804,mon_config_key=3257012160,monmap=999369809,nvmeofgw=598337686,osd_metadata=753924036,osd_pg_creating=1265427177,osd_snap=4020555839,osdmap=554001529}) 2024-09-10T20:32:56.751 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=21,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=15} crc {logm=2117149430,mdsmap=3397941211,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=3035793893,mon_config_key=2599193386,monmap=999369809,nvmeofgw=598337686,osd_metadata=753924036,osd_pg_creating=1265427177,osd_snap=406244889,osdmap=1900427801}) 2024-09-10T20:32:56.751 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.751 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=2,mgr=7,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=19} crc {logm=674639863,mdsmap=3140514009,mgr=1914389234,mgr_metadata=3232925821,mgrstat=4158154804,mon_config_key=3257012160,monmap=999369809,nvmeofgw=598337686,osd_metadata=753924036,osd_pg_creating=1265427177,osd_snap=4020555839,osdmap=554001529}) 2024-09-10T20:32:56.751 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=12} crc {logm=2604561452,mdsmap=3397941211,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=1949580516,mon_config_key=3257012160,monmap=999369809,nvmeofgw=598337686,osd_metadata=2472831596,osd_pg_creating=1265427177,osd_snap=2216181629,osdmap=756917193}) 2024-09-10T20:32:56.753 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.753 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2479603504}) 2024-09-10T20:32:56.753 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=253401472}) 2024-09-10T20:32:56.753 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.753 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2479603504}) 2024-09-10T20:32:56.753 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.746+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1621341763}) 2024-09-10T20:32:56.755 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.755 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3662928976}) 2024-09-10T20:32:56.756 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3213903493}) 2024-09-10T20:32:56.756 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.756 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3662928976}) 2024-09-10T20:32:56.756 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3131793422}) 2024-09-10T20:32:56.758 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.758 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3382487384}) 2024-09-10T20:32:56.758 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2051700695}) 2024-09-10T20:32:56.758 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.758 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3382487384}) 2024-09-10T20:32:56.758 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.750+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3684239183}) 2024-09-10T20:32:56.762 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.754+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.762 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.754+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3462365302}) 2024-09-10T20:32:56.762 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.754+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3264655686}) 2024-09-10T20:32:56.762 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.754+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.762 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.754+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3462365302}) 2024-09-10T20:32:56.762 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.754+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2801873183}) 2024-09-10T20:32:56.780 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.774+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.780 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.774+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=2738291312}) 2024-09-10T20:32:56.780 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.774+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=30} crc {osdmap=3106000279}) 2024-09-10T20:32:56.780 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.774+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:32:56.780 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.774+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=31} crc {osdmap=2738291312}) 2024-09-10T20:32:56.780 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:32:56.774+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=29} crc {osdmap=3283415877}) 2024-09-10T20:32:57.716 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T20:32:57.716 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:32:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient: get_monmap_and_config 2024-09-10T20:32:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient: build_initial_monmap 2024-09-10T20:32:57.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient: monmap: 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:57.847294+0000 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:57.847294+0000 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:57.849 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 1 Processor -- start 2024-09-10T20:32:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 1 -- start start 2024-09-10T20:32:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient: init 2024-09-10T20:32:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-10T20:32:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fed5809ce10 0x7fed5809d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:57.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient(hunting): picked mon.noname-a con 0x7fed5809ce10 addr v2:172.21.15.153:3300/0 2024-09-10T20:32:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fed580a0270 addr v1:172.21.15.176:6789/0 2024-09-10T20:32:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58135c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient(hunting): picked mon.noname-b con 0x7fed5809d730 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fed580a3040 con 0x7fed580a0270 2024-09-10T20:32:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fed5809ea30 con 0x7fed5809ce10 2024-09-10T20:32:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:57.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fed5809f5b0 con 0x7fed5809d730 2024-09-10T20:32:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5f77f640 10 monclient(hunting): authenticate will time out at 1337.986912s 2024-09-10T20:32:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fed5809ce10 0x7fed5809d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58135c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fed5809ce10 0x7fed5809d1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55352/0 (socket says 172.21.15.153:55352) 2024-09-10T20:32:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58135c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46346/0 (socket says 172.21.15.153:46346) 2024-09-10T20:32:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 1 -- 172.21.15.153:0/1362501797 learned_addr learned my addr 172.21.15.153:0/1362501797 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 10 monclient(hunting): get_auth_request con 0x7fed5809ce10 auth_method 0 2024-09-10T20:32:57.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient(hunting): get_auth_request con 0x7fed5809d730 auth_method 0 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:57.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient(hunting): handle_auth_done global_id 44123 payload 274 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 1 -- 172.21.15.153:0/1362501797 >> v2:172.21.15.153:3300/0 conn(0x7fed5809ce10 msgr2=0x7fed5809d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 1 --2- 172.21.15.153:0/1362501797 >> v2:172.21.15.153:3300/0 conn(0x7fed5809ce10 0x7fed5809d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 1 -- 172.21.15.153:0/1362501797 >> v1:172.21.15.176:6789/0 conn(0x7fed580a0270 legacy=0x7fed58135580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient: _finish_hunting 0 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 1 monclient: found mon.noname-b 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5dd1c640 1 -- 172.21.15.153:0/1362501797 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5809f890 con 0x7fed5809d730 2024-09-10T20:32:57.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.842+0000 7fed5d51b640 1 --2- 172.21.15.153:0/1362501797 >> v2:172.21.15.153:3300/0 conn(0x7fed5809ce10 0x7fed5809d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:32:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed5dd1c640 1 -- 172.21.15.153:0/1362501797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 msgr2=0x7fed58135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed5dd1c640 1 --2- 172.21.15.153:0/1362501797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58135c90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fed50009d90 tx=0x7fed5001c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff37329384da32b5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 1 -- 172.21.15.153:0/1362501797 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed500ac070 con 0x7fed5809d730 2024-09-10T20:32:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:32:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 10 monclient: dump: 2024-09-10T20:32:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:57.865 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 10 monclient: _finish_auth 0 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 10 monclient: _check_auth_tickets 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 1 -- 172.21.15.153:0/1362501797 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed500a6680 con 0x7fed5809d730 2024-09-10T20:32:57.866 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 1 -- 172.21.15.153:0/1362501797 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed500a6a50 con 0x7fed5809d730 2024-09-10T20:32:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed4ffff640 10 monclient: dump: 2024-09-10T20:32:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:57.867 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed5f77f640 5 monclient: authenticate success, global_id 44123 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed5f77f640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.846+0000 7fed5f77f640 4 set_mon_vals no callback set 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient: shutdown 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 -- 172.21.15.153:0/1362501797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 msgr2=0x7fed58135c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:57.868 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 --2- 172.21.15.153:0/1362501797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58135c90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fed50009d90 tx=0x7fed5001c830 comp rx=0 tx=0).stop 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 -- 172.21.15.153:0/1362501797 shutdown_connections 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 --2- 172.21.15.153:0/1362501797 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58135c90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 --2- 172.21.15.153:0/1362501797 >> v2:172.21.15.153:3300/0 conn(0x7fed5809ce10 0x7fed5809d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 -- 172.21.15.153:0/1362501797 >> 172.21.15.153:0/1362501797 conn(0x7fed580a71b0 msgr2=0x7fed580a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 -- 172.21.15.153:0/1362501797 shutdown_connections 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 -- 172.21.15.153:0/1362501797 wait complete. 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient: build_initial_monmap 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient: monmap: 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:57.869 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 librados: starting msgr at 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 librados: starting objecter 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 Processor -- start 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 -- start start 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 librados: setting wanted keys 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 librados: calling monclient init 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient: init 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:32:57.870 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed5809ce10 0x7fed58136810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient(hunting): picked mon.noname-a con 0x7fed5809ce10 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58136d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient(hunting): picked mon.noname-c con 0x7fed5809d730 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed58137290 0x7fed581d6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient(hunting): picked mon.noname-b con 0x7fed58137290 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fed58136350 con 0x7fed5809ce10 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fed581361d0 con 0x7fed58137290 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fed5813e060 con 0x7fed5809d730 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5f77f640 10 monclient(hunting): authenticate will time out at 1337.994079s 2024-09-10T20:32:57.871 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed5809ce10 0x7fed58136810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5cd1a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58136d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed5809ce10 0x7fed58136810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55360/0 (socket says 172.21.15.153:55360) 2024-09-10T20:32:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5cd1a640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58136d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46352/0 (socket says 172.21.15.153:46352) 2024-09-10T20:32:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 1 -- 172.21.15.153:0/1578448256 learned_addr learned my addr 172.21.15.153:0/1578448256 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient(hunting): get_auth_request con 0x7fed5809ce10 auth_method 0 2024-09-10T20:32:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:57.872 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5cd1a640 10 monclient(hunting): get_auth_request con 0x7fed5809d730 auth_method 0 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5cd1a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5cd1a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5cd1a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5cd1a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5cd1a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5cd1a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient(hunting): handle_auth_done global_id 34278 payload 824 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 1 -- 172.21.15.153:0/1578448256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 msgr2=0x7fed58136d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 1 --2- 172.21.15.153:0/1578448256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58136d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 1 -- 172.21.15.153:0/1578448256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed58137290 msgr2=0x7fed581d6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:57.873 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5dd1c640 1 --2- 172.21.15.153:0/1578448256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed58137290 0x7fed581d6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 1 --2- 172.21.15.153:0/1578448256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed58137290 0x7fed581d6850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient: _finish_hunting 0 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 1 monclient: found mon.noname-a 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.850+0000 7fed5d51b640 1 -- 172.21.15.153:0/1578448256 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed5813a1b0 con 0x7fed5809ce10 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5d51b640 1 -- 172.21.15.153:0/1578448256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed5809ce10 msgr2=0x7fed58136810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5d51b640 1 --2- 172.21.15.153:0/1578448256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed5809ce10 0x7fed58136810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fed4800e9d0 tx=0x7fed4800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 client.?.objecter ms_handle_connect 0x7fed5809ce10 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 1 -- 172.21.15.153:0/1578448256 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48017070 con 0x7fed5809ce10 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:57.874 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 monclient: dump: 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 monclient: _finish_auth 0 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 monclient: _check_auth_tickets 2024-09-10T20:32:57.875 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 1 -- 172.21.15.153:0/1578448256 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fed4800cc70 con 0x7fed5809ce10 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 5 monclient: authenticate success, global_id 34278 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 10 monclient: _renew_subs 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed58139d90 con 0x7fed5809ce10 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 1 -- 172.21.15.153:0/1578448256 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed48013670 con 0x7fed5809ce10 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 monclient: dump: 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:57.876 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 10 client.34278.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 10 monclient: _renew_subs 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4f7fe640 4 set_mon_vals no callback set 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed58137a30 con 0x7fed5809ce10 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 1 librados: init done 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:57.877 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 1 -- 172.21.15.153:0/1578448256 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fed4801a070 con 0x7fed5809ce10 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed5813e4a0 con 0x7fed5809ce10 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 1 --2- 172.21.15.153:0/1578448256 >> v2:172.21.15.176:6812/3003473398 conn(0x7fed3c03cea0 0x7fed3c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 1 -- 172.21.15.153:0/1578448256 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(299..299 src has 1..299) ==== 5220+0+0 (secure 0 0 0) 0x7fed4804f8b0 con 0x7fed5809ce10 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 10 client.34278.objecter ms_dispatch 0x7fed58000ba0 osd_map(299..299 src has 1..299) 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 3 client.34278.objecter handle_osd_map got epochs [299,299] > 0 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 3 client.34278.objecter handle_osd_map decoding full epoch 299 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed4dffb640 20 client.34278.objecter dump_active .. 0 homeless 2024-09-10T20:32:57.878 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5cd1a640 1 --2- 172.21.15.153:0/1578448256 >> v2:172.21.15.176:6812/3003473398 conn(0x7fed3c03cea0 0x7fed3c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:57.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.854+0000 7fed5cd1a640 10 monclient: get_auth_request con 0x7fed3c03cea0 auth_method 0 2024-09-10T20:32:57.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.858+0000 7fed4dffb640 10 client.34278.objecter ms_handle_connect 0x7fed3c03cea0 2024-09-10T20:32:57.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.858+0000 7fed5cd1a640 1 --2- 172.21.15.153:0/1578448256 >> v2:172.21.15.176:6812/3003473398 conn(0x7fed3c03cea0 0x7fed3c03f360 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fed58138520 tx=0x7fed54007e40 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:57.879 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.858+0000 7fed4dffb640 1 -- 172.21.15.153:0/1578448256 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fed5813a1b0 con 0x7fed5809ce10 2024-09-10T20:32:57.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.858+0000 7fed4dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:57.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:57.858+0000 7fed4dffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:58.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed5f77f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:32:58.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed5f77f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:32:58.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed5f77f640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:58.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fed5806a660 con 0x7fed5809ce10 2024-09-10T20:32:58.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed4dffb640 1 -- 172.21.15.153:0/1578448256 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7fed4800ce20 con 0x7fed5809ce10 2024-09-10T20:32:58.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed4dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:32:58.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed4dffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:58.101 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:32:58.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed5f77f640 10 librados: watch_flush enter 2024-09-10T20:32:58.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed5f77f640 10 librados: watch_flush exit 2024-09-10T20:32:58.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed5f77f640 20 client.34278.objecter shutdown clearing up homeless session... 2024-09-10T20:32:58.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.094+0000 7fed5f77f640 10 client.34278.objecter successfully canceled tick 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 >> v2:172.21.15.176:6812/3003473398 conn(0x7fed3c03cea0 msgr2=0x7fed3c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 --2- 172.21.15.153:0/1578448256 >> v2:172.21.15.176:6812/3003473398 conn(0x7fed3c03cea0 0x7fed3c03f360 secure :-1 s=READY pgs=3351 cs=0 l=1 rev1=1 crypto rx=0x7fed58138520 tx=0x7fed54007e40 comp rx=0 tx=0).stop 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 10 monclient: shutdown 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed5809ce10 msgr2=0x7fed58136810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 --2- 172.21.15.153:0/1578448256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed5809ce10 0x7fed58136810 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fed4800e9d0 tx=0x7fed4800ee90 comp rx=0 tx=0).stop 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 shutdown_connections 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 --2- 172.21.15.153:0/1578448256 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fed58137290 0x7fed581d6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 --2- 172.21.15.153:0/1578448256 >> v2:172.21.15.176:6812/3003473398 conn(0x7fed3c03cea0 0x7fed3c03f360 unknown :-1 s=CLOSED pgs=3351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 --2- 172.21.15.153:0/1578448256 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fed5809d730 0x7fed58136d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 --2- 172.21.15.153:0/1578448256 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fed5809ce10 0x7fed58136810 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 >> 172.21.15.153:0/1578448256 conn(0x7fed580a71b0 msgr2=0x7fed5813cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 shutdown_connections 2024-09-10T20:32:58.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 -- 172.21.15.153:0/1578448256 wait complete. 2024-09-10T20:32:58.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.098+0000 7fed5f77f640 1 librados: shutdown 2024-09-10T20:32:58.118 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T20:32:58.118 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-10T20:32:58.118 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T20:32:58.118 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:32:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient: get_monmap_and_config 2024-09-10T20:32:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient: monmap: 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:58.235139+0000 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:58.235139+0000 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:58.237 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:58.238 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 1 Processor -- start 2024-09-10T20:32:58.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 1 -- start start 2024-09-10T20:32:58.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient: init 2024-09-10T20:32:58.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3b580a41a0 0x7f3b580a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient(hunting): picked mon.noname-c con 0x7f3b580a41a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3b580a4580 addr v1:172.21.15.176:6789/0 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3b580a49d0 addr v1:172.21.15.153:6789/0 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b580a67f0 con 0x7f3b580a49d0 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3b580a6a20 con 0x7f3b580a4580 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3b580a2570 con 0x7f3b580a41a0 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5d231640 10 monclient(hunting): authenticate will time out at 1338.374770s 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5759e640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f3b580a49d0 0x7f3b580a60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37334/0 (socket says 172.21.15.153:37334) 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b5759e640 1 -- 172.21.15.153:0/516762353 learned_addr learned my addr 172.21.15.153:0/516762353 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:32:58.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 1 --2- 172.21.15.153:0/516762353 >> v2:172.21.15.176:3300/0 conn(0x7f3b580a41a0 0x7f3b580a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient(hunting): get_auth_request con 0x7f3b580a41a0 auth_method 0 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient(hunting): handle_auth_done global_id 20134 payload 274 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 1 -- 172.21.15.153:0/516762353 >> v1:172.21.15.176:6789/0 conn(0x7f3b580a4580 legacy=0x7f3b580a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 1 -- 172.21.15.153:0/516762353 >> v1:172.21.15.153:6789/0 conn(0x7f3b580a49d0 legacy=0x7f3b580a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 1 monclient: found mon.noname-c 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:32:58.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.230+0000 7f3b56d9d640 1 -- 172.21.15.153:0/516762353 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b580a3c80 con 0x7f3b580a41a0 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 1 -- 172.21.15.153:0/516762353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b580a41a0 msgr2=0x7f3b580a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 1 --2- 172.21.15.153:0/516762353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b580a41a0 0x7f3b580a3740 secure :-1 s=READY pgs=5721 cs=0 l=1 rev1=1 crypto rx=0x7f3b480097c0 tx=0x7f3b4801c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9b62637b435b5f21 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 1 -- 172.21.15.153:0/516762353 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480b0070 con 0x7f3b580a41a0 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 10 monclient: dump: 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 5 monclient: authenticate success, global_id 20134 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:32:58.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 1 -- 172.21.15.153:0/516762353 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b480ab600 con 0x7f3b580a41a0 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 1 -- 172.21.15.153:0/516762353 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480ab9d0 con 0x7f3b580a41a0 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 4 set_mon_vals no callback set 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b55d9b640 10 monclient: dump: 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.246 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient: shutdown 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 -- 172.21.15.153:0/516762353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b580a41a0 msgr2=0x7f3b580a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 --2- 172.21.15.153:0/516762353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b580a41a0 0x7f3b580a3740 secure :-1 s=READY pgs=5721 cs=0 l=1 rev1=1 crypto rx=0x7f3b480097c0 tx=0x7f3b4801c360 comp rx=0 tx=0).stop 2024-09-10T20:32:58.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 -- 172.21.15.153:0/516762353 shutdown_connections 2024-09-10T20:32:58.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 --2- 172.21.15.153:0/516762353 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b580a41a0 0x7f3b580a3740 unknown :-1 s=CLOSED pgs=5721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 -- 172.21.15.153:0/516762353 >> 172.21.15.153:0/516762353 conn(0x7f3b580ab7e0 msgr2=0x7f3b580a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 -- 172.21.15.153:0/516762353 shutdown_connections 2024-09-10T20:32:58.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 -- 172.21.15.153:0/516762353 wait complete. 2024-09-10T20:32:58.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient: monmap: 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 librados: starting msgr at 2024-09-10T20:32:58.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 librados: starting objecter 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 Processor -- start 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 -- start start 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 librados: setting wanted keys 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 librados: calling monclient init 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient: init 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b580a41a0 0x7f3b58128a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient(hunting): picked mon.noname-c con 0x7f3b580a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b580a4580 0x7f3b58140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient(hunting): picked mon.noname-a con 0x7f3b580a4580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b580a49d0 0x7f3b581d0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient(hunting): picked mon.noname-b con 0x7f3b580a49d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:58.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3b58144630 con 0x7f3b580a4580 2024-09-10T20:32:58.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3b58070400 con 0x7f3b580a49d0 2024-09-10T20:32:58.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3b581447b0 con 0x7f3b580a41a0 2024-09-10T20:32:58.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5d231640 10 monclient(hunting): authenticate will time out at 1338.382704s 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b580a4580 0x7f3b58140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b580a41a0 0x7f3b58128a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b580a4580 0x7f3b58140880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55376/0 (socket says 172.21.15.153:55376) 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 1 -- 172.21.15.153:0/1558252475 learned_addr learned my addr 172.21.15.153:0/1558252475 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 10 monclient(hunting): get_auth_request con 0x7f3b580a4580 auth_method 0 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient(hunting): get_auth_request con 0x7f3b580a41a0 auth_method 0 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.257 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5759e640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b580a49d0 0x7f3b581d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5759e640 10 monclient(hunting): get_auth_request con 0x7f3b580a49d0 auth_method 0 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5759e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5759e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5759e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient(hunting): handle_auth_done global_id 44129 payload 824 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b580a49d0 msgr2=0x7f3b581d0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3b580a49d0 0x7f3b581d0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b580a4580 msgr2=0x7f3b58140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b580a4580 0x7f3b58140880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 1 monclient: found mon.noname-c 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b56d9d640 1 -- 172.21.15.153:0/1558252475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b58129f20 con 0x7f3b580a41a0 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.238+0000 7f3b5659c640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b580a4580 0x7f3b58140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b56d9d640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b580a41a0 msgr2=0x7f3b58128a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b56d9d640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b580a41a0 0x7f3b58128a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3b480ab000 tx=0x7f3b4801aba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 client.?.objecter ms_handle_connect 0x7f3b580a41a0 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480b0050 con 0x7f3b580a41a0 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 monclient: dump: 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 5 monclient: authenticate success, global_id 44129 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3b480bc070 con 0x7f3b580a41a0 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b480aad30 con 0x7f3b580a41a0 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 10 monclient: _renew_subs 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b580a21c0 con 0x7f3b580a41a0 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 monclient: dump: 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.261 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b54d99640 4 set_mon_vals no callback set 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 10 client.44129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 10 monclient: _renew_subs 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b58127ea0 con 0x7f3b580a41a0 2024-09-10T20:32:58.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 1 librados: init done 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581268a0 0x7f3b581d0ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3b5812a160 con 0x7f3b581268a0 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f3b480c1460 con 0x7f3b580a41a0 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581268a0 0x7f3b581d0ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 10 monclient: get_auth_request con 0x7f3b581268a0 auth_method 0 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 10 monclient(hunting): handle_auth_done global_id 34284 payload 824 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5659c640 1 -- 172.21.15.153:0/1558252475 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3b580a3720 con 0x7f3b581268a0 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 1 --2- 172.21.15.153:0/1558252475 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3b4403ced0 0x7f3b4403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(299..299 src has 1..299) ==== 5220+0+0 (secure 0 0 0) 0x7f3b480ea440 con 0x7f3b580a41a0 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 10 client.44129.objecter ms_dispatch 0x7f3b58000ba0 osd_map(299..299 src has 1..299) 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 3 client.44129.objecter handle_osd_map got epochs [299,299] > 0 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 3 client.44129.objecter handle_osd_map decoding full epoch 299 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b3f7fe640 20 client.44129.objecter dump_active .. 0 homeless 2024-09-10T20:32:58.264 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5759e640 1 --2- 172.21.15.153:0/1558252475 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3b4403ced0 0x7f3b4403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.242+0000 7f3b5759e640 10 monclient: get_auth_request con 0x7f3b4403ced0 auth_method 0 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b5659c640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581268a0 msgr2=0x7f3b581d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 10 client.44129.objecter ms_handle_connect 0x7f3b581268a0 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b5659c640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581268a0 0x7f3b581d0ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b580a11c0 tx=0x7f3b4c0045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 10 client.44129.objecter resend_mon_ops 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c018030 con 0x7f3b581268a0 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 10 monclient: dump: 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.265 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3b4c00ddf0 con 0x7f3b581268a0 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:58.266 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581268a0 msgr2=0x7f3b581d0ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581268a0 0x7f3b581d0ed0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b580a11c0 tx=0x7f3b4c0045a0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581268a0 msgr2=0x7f3b581d0ed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:58.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581268a0 0x7f3b581d0ed0 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3b580a11c0 tx=0x7f3b4c0045a0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b3f7fe640 10 client.44129.objecter ms_handle_connect 0x7f3b4403ced0 2024-09-10T20:32:58.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.246+0000 7f3b5759e640 1 --2- 172.21.15.153:0/1558252475 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3b4403ced0 0x7f3b4403f390 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7f3b58128790 tx=0x7f3b50009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.268 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5d231640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5d231640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581d1410 0x7f3b58121620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5d231640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3b581d1820 con 0x7f3b581d1410 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581d1410 0x7f3b58121620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 10 monclient: get_auth_request con 0x7f3b581d1410 auth_method 0 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 10 monclient(hunting): handle_auth_done global_id 34287 payload 824 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.250+0000 7f3b5659c640 1 -- 172.21.15.153:0/1558252475 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3b58127820 con 0x7f3b581d1410 2024-09-10T20:32:58.269 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5659c640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581d1410 msgr2=0x7f3b58121620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5659c640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581d1410 0x7f3b58121620 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c018000 tx=0x7f3b4c007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 10 client.44129.objecter ms_handle_connect 0x7f3b581d1410 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 10 client.44129.objecter resend_mon_ops 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b4c01ee40 con 0x7f3b581d1410 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 10 monclient: dump: 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.270 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f3b4c01e020 con 0x7f3b581d1410 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581d1410 msgr2=0x7f3b58121620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581d1410 0x7f3b58121620 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c018000 tx=0x7f3b4c007f90 comp rx=0 tx=0).stop 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581d1410 msgr2=0x7f3b58121620 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b3f7fe640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3b581d1410 0x7f3b58121620 secure :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3b4c018000 tx=0x7f3b4c007f90 comp rx=0 tx=0).stop 2024-09-10T20:32:58.271 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 32, 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 138814, 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-10T20:32:58.272 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:58.273 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:32:58.273 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:32:58.273 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:58.273 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:58.273 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:58.273 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:58.273 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:58.273 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:58.273 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:58.274 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:32:58.275 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:58.276 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.277 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.278 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.279 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:58.280 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:58.281 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:32:58.282 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.283 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:58.284 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:58.284 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:58.284 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.284 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.284 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.284 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:32:58.284 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 10 librados: watch_flush enter 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 10 librados: watch_flush exit 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 20 client.44129.objecter shutdown clearing up homeless session... 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 10 client.44129.objecter successfully canceled tick 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3b4403ced0 msgr2=0x7f3b4403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 --2- 172.21.15.153:0/1558252475 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3b4403ced0 0x7f3b4403f390 secure :-1 s=READY pgs=3352 cs=0 l=1 rev1=1 crypto rx=0x7f3b58128790 tx=0x7f3b50009040 comp rx=0 tx=0).stop 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 10 monclient: shutdown 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b580a41a0 msgr2=0x7f3b58128a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5759e640 1 -- 172.21.15.153:0/1558252475 reap_dead start 2024-09-10T20:32:58.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b580a41a0 0x7f3b58128a00 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3b480ab000 tx=0x7f3b4801aba0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 shutdown_connections 2024-09-10T20:32:58.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 --2- 172.21.15.153:0/1558252475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3b580a41a0 0x7f3b58128a00 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 >> 172.21.15.153:0/1558252475 conn(0x7f3b580ab7e0 msgr2=0x7f3b580a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 shutdown_connections 2024-09-10T20:32:58.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 -- 172.21.15.153:0/1558252475 wait complete. 2024-09-10T20:32:58.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.254+0000 7f3b5d231640 1 librados: shutdown 2024-09-10T20:32:58.287 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:32:58.287 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T20:32:58.288 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T20:32:58.404 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.394+0000 7fd37f164640 10 monclient: get_monmap_and_config 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.394+0000 7fd37f164640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.394+0000 7fd37f164640 10 monclient: monmap: 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:58.401122+0000 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:58.401122+0000 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:58.405 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.394+0000 7fd37f164640 1 Processor -- start 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 1 -- start start 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 10 monclient: init 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 10 monclient: _add_conns ranks=[4,0,1,2,3] 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd3780a6290 0x7fd3780a4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 10 monclient(hunting): picked mon.noname-c con 0x7fd3780a6290 addr v2:172.21.15.176:3300/0 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd37806f270 addr v1:172.21.15.153:6789/0 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd37813db20 addr v1:172.21.15.176:6789/0 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd37806f6f0 con 0x7fd37806f270 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd378070790 con 0x7fd37813db20 2024-09-10T20:32:58.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd378070a70 con 0x7fd3780a6290 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37f164640 10 monclient(hunting): authenticate will time out at 1338.540693s 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd377fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd37806f270 0x7fd3780a4e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37348/0 (socket says 172.21.15.153:37348) 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd377fff640 1 -- 172.21.15.153:0/3503955026 learned_addr learned my addr 172.21.15.153:0/3503955026 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 1 --2- 172.21.15.153:0/3503955026 >> v2:172.21.15.176:3300/0 conn(0x7fd3780a6290 0x7fd3780a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient(hunting): get_auth_request con 0x7fd3780a6290 auth_method 0 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient(hunting): handle_auth_done global_id 20140 payload 274 2024-09-10T20:32:58.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 1 -- 172.21.15.153:0/3503955026 >> v1:172.21.15.176:6789/0 conn(0x7fd37813db20 legacy=0x7fd37813ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:58.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 1 -- 172.21.15.153:0/3503955026 >> v1:172.21.15.153:6789/0 conn(0x7fd37806f270 legacy=0x7fd3780a4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:58.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 1 monclient: found mon.noname-c 2024-09-10T20:32:58.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:32:58.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.398+0000 7fd37cf00640 1 -- 172.21.15.153:0/3503955026 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd37809fbf0 con 0x7fd3780a6290 2024-09-10T20:32:58.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37cf00640 1 -- 172.21.15.153:0/3503955026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3780a6290 msgr2=0x7fd3780a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37cf00640 1 --2- 172.21.15.153:0/3503955026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3780a6290 0x7fd3780a4950 secure :-1 s=READY pgs=5723 cs=0 l=1 rev1=1 crypto rx=0x7fd3640097c0 tx=0x7fd36401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc1eabd8d2d12a01 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 1 -- 172.21.15.153:0/3503955026 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3640b0070 con 0x7fd3780a6290 2024-09-10T20:32:58.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:32:58.410 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 10 monclient: dump: 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 5 monclient: authenticate success, global_id 20140 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 1 -- 172.21.15.153:0/3503955026 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3640ab600 con 0x7fd3780a6290 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 4 set_mon_vals no callback set 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 1 -- 172.21.15.153:0/3503955026 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3640ab9d0 con 0x7fd3780a6290 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd3777fe640 10 monclient: dump: 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.412 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 10 monclient: shutdown 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 -- 172.21.15.153:0/3503955026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3780a6290 msgr2=0x7fd3780a4950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 --2- 172.21.15.153:0/3503955026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3780a6290 0x7fd3780a4950 secure :-1 s=READY pgs=5723 cs=0 l=1 rev1=1 crypto rx=0x7fd3640097c0 tx=0x7fd36401c360 comp rx=0 tx=0).stop 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 -- 172.21.15.153:0/3503955026 shutdown_connections 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 --2- 172.21.15.153:0/3503955026 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd3780a6290 0x7fd3780a4950 unknown :-1 s=CLOSED pgs=5723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 -- 172.21.15.153:0/3503955026 >> 172.21.15.153:0/3503955026 conn(0x7fd3780a96f0 msgr2=0x7fd3780a5a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.413 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 -- 172.21.15.153:0/3503955026 shutdown_connections 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 -- 172.21.15.153:0/3503955026 wait complete. 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 10 monclient: monmap: 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:58.414 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 librados: starting msgr at 2024-09-10T20:32:58.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 librados: starting objecter 2024-09-10T20:32:58.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 Processor -- start 2024-09-10T20:32:58.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 -- start start 2024-09-10T20:32:58.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 librados: setting wanted keys 2024-09-10T20:32:58.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 1 librados: calling monclient init 2024-09-10T20:32:58.415 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.402+0000 7fd37f164640 10 monclient: init 2024-09-10T20:32:58.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:32:58.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37806f270 0x7fd37813ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient(hunting): picked mon.noname-c con 0x7fd37806f270 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:58.420 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3780a6290 0x7fd37813d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient(hunting): picked mon.noname-a con 0x7fd3780a6290 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd37813db20 0x7fd3781d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient(hunting): picked mon.noname-b con 0x7fd37813db20 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd3781441e0 con 0x7fd3780a6290 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd378144060 con 0x7fd37813db20 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd378144360 con 0x7fd37806f270 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient(hunting): authenticate will time out at 1338.548332s 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37806f270 0x7fd37813ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.421 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3780a6290 0x7fd37813d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37806f270 0x7fd37813ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46382/0 (socket says 172.21.15.153:46382) 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 1 -- 172.21.15.153:0/762817859 learned_addr learned my addr 172.21.15.153:0/762817859 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 10 monclient(hunting): get_auth_request con 0x7fd37806f270 auth_method 0 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3780a6290 0x7fd37813d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55414/0 (socket says 172.21.15.153:55414) 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient(hunting): get_auth_request con 0x7fd3780a6290 auth_method 0 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.422 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37d701640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd37813db20 0x7fd3781d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient(hunting): handle_auth_done global_id 34290 payload 824 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37806f270 msgr2=0x7fd37813ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37806f270 0x7fd37813ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd37813db20 msgr2=0x7fd3781d06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd37813db20 0x7fd3781d06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 monclient: found mon.noname-a 2024-09-10T20:32:58.423 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 -- 172.21.15.153:0/762817859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd378125370 con 0x7fd3780a6290 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37d701640 10 monclient: get_auth_request con 0x7fd37813db20 auth_method 0 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37d701640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd37813db20 0x7fd3781d06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37806f270 0x7fd37813ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3780a6290 msgr2=0x7fd37813d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd377fff640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3780a6290 0x7fd37813d380 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00b4c0 tx=0x7fd36c00b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 client.?.objecter ms_handle_connect 0x7fd3780a6290 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c015020 con 0x7fd3780a6290 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:58.424 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 monclient: dump: 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.425 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd36c011a60 con 0x7fd3780a6290 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd36c011e50 con 0x7fd3780a6290 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 5 monclient: authenticate success, global_id 34290 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient: _renew_subs 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3780a0130 con 0x7fd3780a6290 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3767fc640 4 set_mon_vals no callback set 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd3757fa640 10 monclient: dump: 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.426 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 client.34290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient: _renew_subs 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd378125f20 con 0x7fd3780a6290 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 librados: init done 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802d450 0x7fd37802d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd3780a4950 con 0x7fd37802d450 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.406+0000 7fd37cf00640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802d450 0x7fd37802d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 10 monclient: get_auth_request con 0x7fd37802d450 auth_method 0 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 10 monclient(hunting): handle_auth_done global_id 44135 payload 824 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fd36c018020 con 0x7fd3780a6290 2024-09-10T20:32:58.428 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 1 -- 172.21.15.153:0/762817859 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd37813f610 con 0x7fd37802d450 2024-09-10T20:32:58.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 1 --2- 172.21.15.153:0/762817859 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd35c03d070 0x7fd35c03f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(299..299 src has 1..299) ==== 5220+0+0 (secure 0 0 0) 0x7fd36c04f2c0 con 0x7fd3780a6290 2024-09-10T20:32:58.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 10 client.34290.objecter ms_dispatch 0x7fd378000ba0 osd_map(299..299 src has 1..299) 2024-09-10T20:32:58.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 3 client.34290.objecter handle_osd_map got epochs [299,299] > 0 2024-09-10T20:32:58.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 3 client.34290.objecter handle_osd_map decoding full epoch 299 2024-09-10T20:32:58.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 20 client.34290.objecter dump_active .. 0 homeless 2024-09-10T20:32:58.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37d701640 1 --2- 172.21.15.153:0/762817859 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd35c03d070 0x7fd35c03f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37d701640 10 monclient: get_auth_request con 0x7fd35c03d070 auth_method 0 2024-09-10T20:32:58.429 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802d450 msgr2=0x7fd37802d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37cf00640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802d450 0x7fd37802d830 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd36401c190 tx=0x7fd364019100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 10 client.34290.objecter ms_handle_connect 0x7fd37802d450 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 10 client.34290.objecter resend_mon_ops 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3640bc070 con 0x7fd37802d450 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 10 monclient: dump: 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.430 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd3640b1040 con 0x7fd37802d450 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802d450 msgr2=0x7fd37802d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802d450 0x7fd37802d830 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd36401c190 tx=0x7fd364019100 comp rx=0 tx=0).stop 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802d450 msgr2=0x7fd37802d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802d450 0x7fd37802d830 secure :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fd36401c190 tx=0x7fd364019100 comp rx=0 tx=0).stop 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd3757fa640 10 client.34290.objecter ms_handle_connect 0x7fd35c03d070 2024-09-10T20:32:58.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.410+0000 7fd37d701640 1 --2- 172.21.15.153:0/762817859 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd35c03d070 0x7fd35c03f530 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7fd370007fd0 tx=0x7fd370009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37f164640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:32:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37f164640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802dd70 0x7fd37802e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37f164640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd37806cd40 con 0x7fd37802dd70 2024-09-10T20:32:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802dd70 0x7fd37802e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 10 monclient: get_auth_request con 0x7fd37802dd70 auth_method 0 2024-09-10T20:32:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 10 monclient(hunting): handle_auth_done global_id 44138 payload 824 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 1 -- 172.21.15.153:0/762817859 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd378123df0 con 0x7fd37802dd70 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802dd70 msgr2=0x7fd37802e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd37cf00640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802dd70 0x7fd37802e180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd36401c3a0 tx=0x7fd364009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd3757fa640 10 client.34290.objecter ms_handle_connect 0x7fd37802dd70 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd3757fa640 10 client.34290.objecter resend_mon_ops 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3640b0040 con 0x7fd37802dd70 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd3757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd3757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd3757fa640 10 monclient: dump: 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.435 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fd364003b20 con 0x7fd37802dd70 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd3757fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.418+0000 7fd3757fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802dd70 msgr2=0x7fd37802e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd3757fa640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802dd70 0x7fd37802e180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd36401c3a0 tx=0x7fd364009190 comp rx=0 tx=0).stop 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd3757fa640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802dd70 msgr2=0x7fd37802e180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:58.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd3757fa640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd37802dd70 0x7fd37802e180 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd36401c3a0 tx=0x7fd364009190 comp rx=0 tx=0).stop 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 32, 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 17871, 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.437 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 2, 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:58.438 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:58.439 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:32:58.440 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:58.441 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:58.442 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:58.443 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:58.444 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:32:58.445 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.446 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:58.447 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:32:58.448 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:32:58.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 10 librados: watch_flush enter 2024-09-10T20:32:58.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 10 librados: watch_flush exit 2024-09-10T20:32:58.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 20 client.34290.objecter shutdown clearing up homeless session... 2024-09-10T20:32:58.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 10 client.34290.objecter successfully canceled tick 2024-09-10T20:32:58.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd35c03d070 msgr2=0x7fd35c03f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 1 --2- 172.21.15.153:0/762817859 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd35c03d070 0x7fd35c03f530 secure :-1 s=READY pgs=3353 cs=0 l=1 rev1=1 crypto rx=0x7fd370007fd0 tx=0x7fd370009040 comp rx=0 tx=0).stop 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 10 monclient: shutdown 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3780a6290 msgr2=0x7fd37813d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 1 --2- 172.21.15.153:0/762817859 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd3780a6290 0x7fd37813d380 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fd36c00b4c0 tx=0x7fd36c00b980 comp rx=0 tx=0).stop 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37d701640 1 -- 172.21.15.153:0/762817859 reap_dead start 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37d701640 1 -- 172.21.15.153:0/762817859 reap_dead start 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 shutdown_connections 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 >> 172.21.15.153:0/762817859 conn(0x7fd3780a96f0 msgr2=0x7fd37809ea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 shutdown_connections 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 1 -- 172.21.15.153:0/762817859 wait complete. 2024-09-10T20:32:58.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.422+0000 7fd37f164640 1 librados: shutdown 2024-09-10T20:32:58.452 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:32:58.452 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T20:32:58.452 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient: get_monmap_and_config 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient: monmap: 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:58.566162+0000 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:58.566162+0000 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.567 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:58.568 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:58.568 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:58.568 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:58.568 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:58.568 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 1 Processor -- start 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 1 -- start start 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient: init 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient: _add_conns ranks=[0,4,2,3,1] 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f336006d5d0 addr v1:172.21.15.153:6789/0 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f33600a2770 0x7f336013da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient(hunting): picked mon.noname-c con 0x7f33600a2770 addr v2:172.21.15.176:3300/0 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2b80 0x7f336013ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient(hunting): picked mon.noname-b con 0x7f33600a2b80 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3360074880 con 0x7f336006d5d0 2024-09-10T20:32:58.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f336006d9b0 con 0x7f33600a2770 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f336009f8d0 con 0x7f33600a2b80 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3366a34640 10 monclient(hunting): authenticate will time out at 1338.705993s 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f335ffff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f336006d5d0 0x7f33600a2060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:37354/0 (socket says 172.21.15.153:37354) 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2b80 0x7f336013ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f335ffff640 1 -- 172.21.15.153:0/2076669192 learned_addr learned my addr 172.21.15.153:0/2076669192 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient(hunting): get_auth_request con 0x7f33600a2b80 auth_method 0 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient(hunting): handle_auth_done global_id 44141 payload 274 2024-09-10T20:32:58.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 1 -- 172.21.15.153:0/2076669192 >> v2:172.21.15.176:3300/0 conn(0x7f33600a2770 msgr2=0x7f336013da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:58.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 1 --2- 172.21.15.153:0/2076669192 >> v2:172.21.15.176:3300/0 conn(0x7f33600a2770 0x7f336013da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 1 -- 172.21.15.153:0/2076669192 >> v1:172.21.15.153:6789/0 conn(0x7f336006d5d0 legacy=0x7f33600a2060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:58.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 1 monclient: found mon.noname-b 2024-09-10T20:32:58.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.572 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.562+0000 7f3364fd1640 1 -- 172.21.15.153:0/2076669192 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336009eae0 con 0x7f33600a2b80 2024-09-10T20:32:58.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3364fd1640 1 -- 172.21.15.153:0/2076669192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2b80 msgr2=0x7f336013ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3364fd1640 1 --2- 172.21.15.153:0/2076669192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2b80 0x7f336013ff80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3358009d90 tx=0x7f335801c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3b2dc9c389b4461 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 1 -- 172.21.15.153:0/2076669192 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580ac070 con 0x7f33600a2b80 2024-09-10T20:32:58.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 10 monclient: dump: 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.574 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 5 monclient: authenticate success, global_id 44141 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 1 -- 172.21.15.153:0/2076669192 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33580a6740 con 0x7f33600a2b80 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 1 -- 172.21.15.153:0/2076669192 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33580a6b10 con 0x7f33600a2b80 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 4 set_mon_vals no callback set 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f335effd640 10 monclient: dump: 2024-09-10T20:32:58.575 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.576 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.576 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 10 monclient: shutdown 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 1 -- 172.21.15.153:0/2076669192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2b80 msgr2=0x7f336013ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 1 --2- 172.21.15.153:0/2076669192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2b80 0x7f336013ff80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3358009d90 tx=0x7f335801c7d0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 1 -- 172.21.15.153:0/2076669192 shutdown_connections 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 1 --2- 172.21.15.153:0/2076669192 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2b80 0x7f336013ff80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 1 --2- 172.21.15.153:0/2076669192 >> v2:172.21.15.176:3300/0 conn(0x7f33600a2770 0x7f336013da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 1 -- 172.21.15.153:0/2076669192 >> 172.21.15.153:0/2076669192 conn(0x7f33600a7830 msgr2=0x7f33600a7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 1 -- 172.21.15.153:0/2076669192 shutdown_connections 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.566+0000 7f3366a34640 1 -- 172.21.15.153:0/2076669192 wait complete. 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient: monmap: 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.582 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 librados: starting msgr at 2024-09-10T20:32:58.583 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 librados: starting objecter 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 Processor -- start 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 -- start start 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 librados: setting wanted keys 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 librados: calling monclient init 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient: init 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336006d5d0 0x7f33601d0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient(hunting): picked mon.noname-b con 0x7f336006d5d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2770 0x7f3360134880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient(hunting): picked mon.noname-c con 0x7f33600a2770 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33600a2b80 0x7f3360134dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.584 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient(hunting): picked mon.noname-a con 0x7f33600a2b80 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3360143070 con 0x7f33600a2b80 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3360142ef0 con 0x7f336006d5d0 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f33601431f0 con 0x7f33600a2770 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 10 monclient(hunting): authenticate will time out at 1338.713032s 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2770 0x7f3360134880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3364fd1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33600a2b80 0x7f3360134dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2770 0x7f3360134880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46418/0 (socket says 172.21.15.153:46418) 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 -- 172.21.15.153:0/1244637481 learned_addr learned my addr 172.21.15.153:0/1244637481 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient(hunting): get_auth_request con 0x7f33600a2770 auth_method 0 2024-09-10T20:32:58.585 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3364fd1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33600a2b80 0x7f3360134dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55424/0 (socket says 172.21.15.153:55424) 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3364fd1640 10 monclient(hunting): get_auth_request con 0x7f33600a2b80 auth_method 0 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3364fd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3364fd1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3364fd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3364fd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3364fd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3364fd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.586 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335ffff640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336006d5d0 0x7f33601d0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient(hunting): handle_auth_done global_id 44144 payload 824 2024-09-10T20:32:58.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336006d5d0 msgr2=0x7f33601d0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336006d5d0 0x7f33601d0530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33600a2b80 msgr2=0x7f3360134dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f33600a2b80 0x7f3360134dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 monclient: found mon.noname-c 2024-09-10T20:32:58.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.587 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 -- 172.21.15.153:0/1244637481 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f336013d5f0 con 0x7f33600a2770 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2770 msgr2=0x7f3360134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335f7fe640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2770 0x7f3360134880 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f335400b410 tx=0x7f335400b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 client.?.objecter ms_handle_connect 0x7f33600a2770 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354015020 con 0x7f33600a2770 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 monclient: dump: 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.588 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f3366a34640 5 monclient: authenticate success, global_id 44144 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3354011a60 con 0x7f33600a2770 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3354011e50 con 0x7f33600a2770 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335cff9640 10 monclient: dump: 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.590 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.570+0000 7f335e7fc640 4 set_mon_vals no callback set 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 10 monclient: _renew_subs 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f336013f950 con 0x7f33600a2770 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 10 client.44144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 10 monclient: _renew_subs 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f33601d0be0 con 0x7f33600a2770 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 1 librados: init done 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013bd40 0x7f336013c120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.591 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f336009e650 con 0x7f336013bd40 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f3354018020 con 0x7f33600a2770 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335cff9640 1 --2- 172.21.15.153:0/1244637481 >> v2:172.21.15.176:6812/3003473398 conn(0x7f334c03d040 0x7f334c03f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013bd40 0x7f336013c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(300..300 src has 1..300) ==== 5220+0+0 (secure 0 0 0) 0x7f335404f790 con 0x7f33600a2770 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335cff9640 10 client.44144.objecter ms_dispatch 0x7f3360000ba0 osd_map(300..300 src has 1..300) 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335cff9640 3 client.44144.objecter handle_osd_map got epochs [300,300] > 0 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335cff9640 3 client.44144.objecter handle_osd_map decoding full epoch 300 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 10 monclient: get_auth_request con 0x7f336013bd40 auth_method 0 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335cff9640 20 client.44144.objecter dump_active .. 0 homeless 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3364fd1640 1 --2- 172.21.15.153:0/1244637481 >> v2:172.21.15.176:6812/3003473398 conn(0x7f334c03d040 0x7f334c03f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f3364fd1640 10 monclient: get_auth_request con 0x7f334c03d040 auth_method 0 2024-09-10T20:32:58.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 10 monclient(hunting): handle_auth_done global_id 20143 payload 824 2024-09-10T20:32:58.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.574+0000 7f335ffff640 1 -- 172.21.15.153:0/1244637481 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33600a2060 con 0x7f336013bd40 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335ffff640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013bd40 msgr2=0x7f336013c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 10 client.44144.objecter ms_handle_connect 0x7f336013bd40 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 10 client.44144.objecter resend_mon_ops 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335ffff640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013bd40 0x7f336013c120 secure :-1 s=READY pgs=5725 cs=0 l=1 rev1=1 crypto rx=0x7f335000c420 tx=0x7f335000c8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 10 client.44144.objecter ms_handle_connect 0x7f334c03d040 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f3364fd1640 1 --2- 172.21.15.153:0/1244637481 >> v2:172.21.15.176:6812/3003473398 conn(0x7f334c03d040 0x7f334c03f500 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f336006fc40 tx=0x7f33580a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3350015070 con 0x7f336013bd40 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 10 monclient: dump: 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.596 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3350011a10 con 0x7f336013bd40 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013bd40 msgr2=0x7f336013c120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013bd40 0x7f336013c120 secure :-1 s=READY pgs=5725 cs=0 l=1 rev1=1 crypto rx=0x7f335000c420 tx=0x7f335000c8e0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013bd40 msgr2=0x7f336013c120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.578+0000 7f335cff9640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013bd40 0x7f336013c120 unknown :-1 s=CLOSED pgs=5725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f3366a34640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f3366a34640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013c660 0x7f3360137970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f3366a34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f336006cd40 con 0x7f336013c660 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f335ffff640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013c660 0x7f3360137970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f335ffff640 10 monclient: get_auth_request con 0x7f336013c660 auth_method 0 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f335ffff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f335ffff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f335ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.582+0000 7f335ffff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335ffff640 10 monclient(hunting): handle_auth_done global_id 20146 payload 824 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335ffff640 1 -- 172.21.15.153:0/1244637481 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f336013e150 con 0x7f336013c660 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335ffff640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013c660 msgr2=0x7f3360137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335ffff640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013c660 0x7f3360137970 secure :-1 s=READY pgs=5726 cs=0 l=1 rev1=1 crypto rx=0x7f335000c420 tx=0x7f33500058e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 10 client.44144.objecter ms_handle_connect 0x7f336013c660 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 10 client.44144.objecter resend_mon_ops 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3350011e00 con 0x7f336013c660 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 10 monclient: dump: 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.599 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f3350011e00 con 0x7f336013c660 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013c660 msgr2=0x7f3360137970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013c660 0x7f3360137970 secure :-1 s=READY pgs=5726 cs=0 l=1 rev1=1 crypto rx=0x7f335000c420 tx=0x7f33500058e0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013c660 msgr2=0x7f3360137970 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:58.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f335cff9640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f336013c660 0x7f3360137970 unknown :-1 s=CLOSED pgs=5726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 32, 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 425233, 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.601 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:58.602 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:58.603 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:32:58.604 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:32:58.605 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.606 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.607 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:32:58.608 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:32:58.609 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:32:58.610 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.611 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:32:58.612 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:32:58.613 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:32:58.613 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:32:58.613 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:32:58.613 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:32:58.613 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:32:58.613 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:32:58.613 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:32:58.613 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:32:58.613 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 10 librados: watch_flush enter 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 10 librados: watch_flush exit 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 20 client.44144.objecter shutdown clearing up homeless session... 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 10 client.44144.objecter successfully canceled tick 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 >> v2:172.21.15.176:6812/3003473398 conn(0x7f334c03d040 msgr2=0x7f334c03f500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 1 --2- 172.21.15.153:0/1244637481 >> v2:172.21.15.176:6812/3003473398 conn(0x7f334c03d040 0x7f334c03f500 secure :-1 s=READY pgs=3354 cs=0 l=1 rev1=1 crypto rx=0x7f336006fc40 tx=0x7f33580a7000 comp rx=0 tx=0).stop 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 10 monclient: shutdown 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2770 msgr2=0x7f3360134880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3366a34640 1 --2- 172.21.15.153:0/1244637481 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f33600a2770 0x7f3360134880 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f335400b410 tx=0x7f335400b8d0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.586+0000 7f3364fd1640 1 -- 172.21.15.153:0/1244637481 reap_dead start 2024-09-10T20:32:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.590+0000 7f3364fd1640 1 -- 172.21.15.153:0/1244637481 reap_dead start 2024-09-10T20:32:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.590+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 shutdown_connections 2024-09-10T20:32:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.590+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 >> 172.21.15.153:0/1244637481 conn(0x7f33600a7830 msgr2=0x7f3360135620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.590+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 shutdown_connections 2024-09-10T20:32:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.590+0000 7f3366a34640 1 -- 172.21.15.153:0/1244637481 wait complete. 2024-09-10T20:32:58.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.590+0000 7f3366a34640 1 librados: shutdown 2024-09-10T20:32:58.616 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:32:58.616 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-10T20:32:58.616 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-10T20:32:58.617 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-10T20:32:58.617 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c store 2024-09-10T20:32:58.617 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c sync_force --yes-i-really-mean-it 2024-09-10T20:32:58.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient: get_monmap_and_config 2024-09-10T20:32:58.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.727 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient: monmap: 2024-09-10T20:32:58.727 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.727 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.727 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:58.726140+0000 2024-09-10T20:32:58.727 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:58.726140+0000 2024-09-10T20:32:58.727 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.728 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.728 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:58.728 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:58.728 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:58.728 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:58.728 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:58.728 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 1 Processor -- start 2024-09-10T20:32:58.728 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 1 -- start start 2024-09-10T20:32:58.729 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient: init 2024-09-10T20:32:58.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient: _add_conns ranks=[4,1,2,0,3] 2024-09-10T20:32:58.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f92340a4140 0x7f92340a38f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient(hunting): picked mon.noname-c con 0x7f92340a4140 addr v2:172.21.15.176:3300/0 2024-09-10T20:32:58.736 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f92340a4520 addr v1:172.21.15.176:6789/0 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f92340a6150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient(hunting): picked mon.noname-b con 0x7f92340a4990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92340a5250 con 0x7f92340a4520 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f92340a2f10 con 0x7f92340a4140 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9234070450 con 0x7f92340a4990 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f923b8dc640 10 monclient(hunting): authenticate will time out at 1338.865841s 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f92340a6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f92340a6150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46430/0 (socket says 172.21.15.153:46430) 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 1 -- 172.21.15.153:0/2760672929 learned_addr learned my addr 172.21.15.153:0/2760672929 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:58.737 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient(hunting): get_auth_request con 0x7f92340a4990 auth_method 0 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239678640 1 --2- 172.21.15.153:0/2760672929 >> v2:172.21.15.176:3300/0 conn(0x7f92340a4140 0x7f92340a38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239678640 10 monclient(hunting): get_auth_request con 0x7f92340a4140 auth_method 0 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239678640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239678640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239678640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient(hunting): handle_auth_done global_id 44147 payload 274 2024-09-10T20:32:58.738 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 1 -- 172.21.15.153:0/2760672929 >> v2:172.21.15.176:3300/0 conn(0x7f92340a4140 msgr2=0x7f92340a38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 1 --2- 172.21.15.153:0/2760672929 >> v2:172.21.15.176:3300/0 conn(0x7f92340a4140 0x7f92340a38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 1 -- 172.21.15.153:0/2760672929 >> v1:172.21.15.176:6789/0 conn(0x7f92340a4520 legacy=0x7f92340a5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 1 monclient: found mon.noname-b 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.722+0000 7f9239e79640 1 -- 172.21.15.153:0/2760672929 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92340a6b00 con 0x7f92340a4990 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f9239e79640 1 -- 172.21.15.153:0/2760672929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 msgr2=0x7f92340a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f9239e79640 1 --2- 172.21.15.153:0/2760672929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f92340a6150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9234078760 tx=0x7f923001c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee2dca0647a17fd4 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 1 -- 172.21.15.153:0/2760672929 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92300b0070 con 0x7f92340a4990 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 10 monclient: dump: 2024-09-10T20:32:58.739 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.740 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 1 -- 172.21.15.153:0/2760672929 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92300ab600 con 0x7f92340a4990 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 5 monclient: authenticate success, global_id 44147 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 1 -- 172.21.15.153:0/2760672929 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92300ab9d0 con 0x7f92340a4990 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 4 set_mon_vals no callback set 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f922bfff640 10 monclient: dump: 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.741 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 10 monclient: shutdown 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 1 -- 172.21.15.153:0/2760672929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 msgr2=0x7f92340a6150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 1 --2- 172.21.15.153:0/2760672929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f92340a6150 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9234078760 tx=0x7f923001c3e0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 1 -- 172.21.15.153:0/2760672929 shutdown_connections 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 1 --2- 172.21.15.153:0/2760672929 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f92340a6150 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 1 --2- 172.21.15.153:0/2760672929 >> v2:172.21.15.176:3300/0 conn(0x7f92340a4140 0x7f92340a38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.742 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 1 -- 172.21.15.153:0/2760672929 >> 172.21.15.153:0/2760672929 conn(0x7f92340a7b40 msgr2=0x7f92340a7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 1 -- 172.21.15.153:0/2760672929 shutdown_connections 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 1 -- 172.21.15.153:0/2760672929 wait complete. 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.726+0000 7f923b8dc640 10 monclient: monmap: 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:58.743 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 librados: starting msgr at 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 librados: starting objecter 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 Processor -- start 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 -- start start 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 librados: setting wanted keys 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 librados: calling monclient init 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient: init 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340a4140 0x7f9234126d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient(hunting): picked mon.noname-a con 0x7f92340a4140 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92340a4520 0x7f9234128d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient(hunting): picked mon.noname-b con 0x7f92340a4520 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:58.744 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f923413fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient(hunting): picked mon.noname-c con 0x7f92340a4990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f9234143070 con 0x7f92340a4140 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f92340a68a0 con 0x7f92340a4520 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f92341431f0 con 0x7f92340a4990 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient(hunting): authenticate will time out at 1338.872717s 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f923413fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340a4140 0x7f9234126d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f923413fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:46432/0 (socket says 172.21.15.153:46432) 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340a4140 0x7f9234126d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55434/0 (socket says 172.21.15.153:55434) 2024-09-10T20:32:58.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 -- 172.21.15.153:0/4133663840 learned_addr learned my addr 172.21.15.153:0/4133663840 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient(hunting): get_auth_request con 0x7f92340a4990 auth_method 0 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 10 monclient(hunting): get_auth_request con 0x7f92340a4140 auth_method 0 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.746 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9238e77640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92340a4520 0x7f9234128d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient(hunting): handle_auth_done global_id 44150 payload 824 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92340a4520 msgr2=0x7f9234128d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f92340a4520 0x7f9234128d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340a4140 msgr2=0x7f9234126d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340a4140 0x7f9234126d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 monclient: found mon.noname-c 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 -- 172.21.15.153:0/4133663840 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92341281e0 con 0x7f92340a4990 2024-09-10T20:32:58.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239678640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f92340a4140 0x7f9234126d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:32:58.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 msgr2=0x7f923413fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9239e79640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f923413fbb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f92300ae040 tx=0x7f9230019180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 client.?.objecter ms_handle_connect 0x7f92340a4990 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92300b0040 con 0x7f92340a4990 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 monclient: dump: 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.752 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92300b4420 con 0x7f92340a4990 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 5 monclient: authenticate success, global_id 44150 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient: _renew_subs 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92340a2230 con 0x7f92340a4990 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.753 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92300b4810 con 0x7f92340a4990 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f9229ffb640 10 monclient: dump: 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f922b7fe640 4 set_mon_vals no callback set 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 client.44150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient: _renew_subs 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9234129f80 con 0x7f92340a4990 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.730+0000 7f923b8dc640 1 librados: init done 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f923b8dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f923b8dc640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92341400f0 0x7f92341d0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f923b8dc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f92341292e0 con 0x7f92341400f0 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92341400f0 0x7f92341d0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 10 monclient: get_auth_request con 0x7f92341400f0 auth_method 0 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.755 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f92300b49f0 con 0x7f92340a4990 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 10 monclient(hunting): handle_auth_done global_id 44153 payload 824 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 1 -- 172.21.15.153:0/4133663840 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92340a38f0 con 0x7f92341400f0 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 1 --2- 172.21.15.153:0/4133663840 >> v2:172.21.15.176:6812/3003473398 conn(0x7f921803cef0 0x7f921803f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(300..300 src has 1..300) ==== 5220+0+0 (secure 0 0 0) 0x7f92300e9440 con 0x7f92340a4990 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 10 client.44150.objecter ms_dispatch 0x7f9234000ba0 osd_map(300..300 src has 1..300) 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 3 client.44150.objecter handle_osd_map got epochs [300,300] > 0 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 3 client.44150.objecter handle_osd_map decoding full epoch 300 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 20 client.44150.objecter dump_active .. 0 homeless 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9238e77640 1 --2- 172.21.15.153:0/4133663840 >> v2:172.21.15.176:6812/3003473398 conn(0x7f921803cef0 0x7f921803f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.756 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9238e77640 10 monclient: get_auth_request con 0x7f921803cef0 auth_method 0 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 10 client.44150.objecter ms_handle_connect 0x7f92341400f0 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92341400f0 msgr2=0x7f92341d0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 10 client.44150.objecter resend_mon_ops 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9239678640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92341400f0 0x7f92341d0d60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9224010b90 tx=0x7f922400e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224018020 con 0x7f92341400f0 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 10 monclient: dump: 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.757 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9224010e90 con 0x7f92341400f0 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92341400f0 msgr2=0x7f92341d0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92341400f0 0x7f92341d0d60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9224010b90 tx=0x7f922400e5a0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92341400f0 msgr2=0x7f92341d0d60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.734+0000 7f9229ffb640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92341400f0 0x7f92341d0d60 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9224010b90 tx=0x7f922400e5a0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.738+0000 7f9229ffb640 10 client.44150.objecter ms_handle_connect 0x7f921803cef0 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.738+0000 7f9238e77640 1 --2- 172.21.15.153:0/4133663840 >> v2:172.21.15.176:6812/3003473398 conn(0x7f921803cef0 0x7f921803f3b0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f922c004450 tx=0x7f922c009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f923b8dc640 10 start_mon_command cmd=[{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f923b8dc640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f923402d450 0x7f923402d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f923b8dc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f9234122da0 con 0x7f923402d450 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f923402d450 0x7f923402d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 10 monclient: get_auth_request con 0x7f923402d450 auth_method 0 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 10 monclient(hunting): handle_auth_done global_id 44156 payload 824 2024-09-10T20:32:58.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.742+0000 7f9239678640 1 -- 172.21.15.153:0/4133663840 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "sync_force", "yes_i_really_mean_it": true}) -- 0x7f923412a340 con 0x7f923402d450 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9239678640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f923402d450 msgr2=0x7f923402d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 10 client.44150.objecter ms_handle_connect 0x7f923402d450 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 10 client.44150.objecter resend_mon_ops 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9239678640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f923402d450 0x7f923402d830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9224019000 tx=0x7f92240046d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9224018030 con 0x7f923402d450 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 10 monclient: dump: 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.761 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+83 (secure 0 0 0) 0x7f9224003830 con 0x7f923402d450 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 10 monclient: handle_command_reply 2 [{"prefix": "sync_force", "yes_i_really_mean_it": true}] 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f923402d450 msgr2=0x7f923402d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f923402d450 0x7f923402d830 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9224019000 tx=0x7f92240046d0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f923402d450 msgr2=0x7f923402d830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:32:58.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9229ffb640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f923402d450 0x7f923402d830 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9224019000 tx=0x7f92240046d0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.763 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:32:58.763 INFO:teuthology.orchestra.run.smithi153.stdout: "ret": 0, 2024-09-10T20:32:58.763 INFO:teuthology.orchestra.run.smithi153.stdout: "msg": "forcing store sync the next time the monitor starts" 2024-09-10T20:32:58.763 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:32:58.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 10 librados: watch_flush enter 2024-09-10T20:32:58.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 10 librados: watch_flush exit 2024-09-10T20:32:58.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 20 client.44150.objecter shutdown clearing up homeless session... 2024-09-10T20:32:58.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 10 client.44150.objecter successfully canceled tick 2024-09-10T20:32:58.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 >> v2:172.21.15.176:6812/3003473398 conn(0x7f921803cef0 msgr2=0x7f921803f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 --2- 172.21.15.153:0/4133663840 >> v2:172.21.15.176:6812/3003473398 conn(0x7f921803cef0 0x7f921803f3b0 secure :-1 s=READY pgs=3355 cs=0 l=1 rev1=1 crypto rx=0x7f922c004450 tx=0x7f922c009040 comp rx=0 tx=0).stop 2024-09-10T20:32:58.764 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 10 monclient: shutdown 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 msgr2=0x7f923413fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f9239e79640 1 -- 172.21.15.153:0/4133663840 reap_dead start 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f923413fbb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f92300ae040 tx=0x7f9230019180 comp rx=0 tx=0).stop 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 shutdown_connections 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 --2- 172.21.15.153:0/4133663840 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f92340a4990 0x7f923413fbb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 >> 172.21.15.153:0/4133663840 conn(0x7f92340a7b40 msgr2=0x7f92340a9760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 shutdown_connections 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 -- 172.21.15.153:0/4133663840 wait complete. 2024-09-10T20:32:58.765 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.746+0000 7f923b8dc640 1 librados: shutdown 2024-09-10T20:32:58.767 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-10T20:32:58.768 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T20:32:58.768 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:32:58.777 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T20:32:58.777 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-10T20:32:58.778 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:32:58.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient: get_monmap_and_config 2024-09-10T20:32:58.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient: monmap: 2024-09-10T20:32:58.932 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:32:58.931411+0000 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:32:58.931411+0000 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:32:58.933 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 1 Processor -- start 2024-09-10T20:32:58.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 1 -- start start 2024-09-10T20:32:58.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient: init 2024-09-10T20:32:58.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient: _add_conns ranks=[2,3,1,0,4] 2024-09-10T20:32:58.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc1364f0 0x7f7bfc1368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient(hunting): picked mon.noname-b con 0x7f7bfc1364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:58.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7bfc1360a0 0x7f7bfc09ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient(hunting): picked mon.noname-a con 0x7f7bfc1360a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:32:58.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7bfc0a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:32:58.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bfc0a27a0 con 0x7f7bfc0a1990 2024-09-10T20:32:58.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f7bfc0a2a80 con 0x7f7bfc1360a0 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7bfc0a2d60 con 0x7f7bfc1364f0 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7c01e34640 10 monclient(hunting): authenticate will time out at 1339.071134s 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfb7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc1364f0 msgr2=0x7f7bfc1368d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc1364f0 0x7f7bfc1368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7bfc1360a0 0x7f7bfc09ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f7bfc1360a0 0x7f7bfc09ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55442/0 (socket says 172.21.15.153:55442) 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 1 -- 172.21.15.153:0/219528492 learned_addr learned my addr 172.21.15.153:0/219528492 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient(hunting): get_auth_request con 0x7f7bfc1360a0 auth_method 0 2024-09-10T20:32:58.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient(hunting): handle_auth_done global_id 34299 payload 274 2024-09-10T20:32:58.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 1 -- 172.21.15.153:0/219528492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc1364f0 msgr2=0x7f7bfc1368d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:32:58.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 1 --2- 172.21.15.153:0/219528492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc1364f0 0x7f7bfc1368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 1 -- 172.21.15.153:0/219528492 >> v1:172.21.15.176:6789/0 conn(0x7f7bfc0a1990 legacy=0x7f7bfc09d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:32:58.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 1 monclient: found mon.noname-a 2024-09-10T20:32:58.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:58.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.926+0000 7f7bfaffd640 1 -- 172.21.15.153:0/219528492 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bfc136e10 con 0x7f7bfc1360a0 2024-09-10T20:32:58.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfaffd640 1 -- 172.21.15.153:0/219528492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1360a0 msgr2=0x7f7bfc09ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfaffd640 1 --2- 172.21.15.153:0/219528492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1360a0 0x7f7bfc09ce10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009870 tx=0x7f7bec01c4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4475dfac9b893a2f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 1 -- 172.21.15.153:0/219528492 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0b0070 con 0x7f7bfc1360a0 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 10 monclient: dump: 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.946 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7c01e34640 5 monclient: authenticate success, global_id 34299 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 1 -- 172.21.15.153:0/219528492 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bec0ab600 con 0x7f7bfc1360a0 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7c01e34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7c01e34640 10 monclient: get_monmap_and_config success 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7c01e34640 4 set_mon_vals no callback set 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 1 -- 172.21.15.153:0/219528492 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bec0ab9d0 con 0x7f7bfc1360a0 2024-09-10T20:32:58.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.930+0000 7f7bfa7fc640 10 monclient: dump: 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient: shutdown 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 -- 172.21.15.153:0/219528492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1360a0 msgr2=0x7f7bfc09ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 --2- 172.21.15.153:0/219528492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1360a0 0x7f7bfc09ce10 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f7bec009870 tx=0x7f7bec01c4d0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 -- 172.21.15.153:0/219528492 shutdown_connections 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 --2- 172.21.15.153:0/219528492 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1360a0 0x7f7bfc09ce10 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 --2- 172.21.15.153:0/219528492 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc1364f0 0x7f7bfc1368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 -- 172.21.15.153:0/219528492 >> 172.21.15.153:0/219528492 conn(0x7f7bfc0a71b0 msgr2=0x7f7bfc0a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 -- 172.21.15.153:0/219528492 shutdown_connections 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 -- 172.21.15.153:0/219528492 wait complete. 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient: build_initial_monmap 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient: monmap: 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:32:58.949 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 librados: starting msgr at 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 librados: starting objecter 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 Processor -- start 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 -- start start 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 librados: setting wanted keys 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 librados: calling monclient init 2024-09-10T20:32:58.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient: init 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient: _reopen_session rank -1 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc0a1990 0x7f7bfc12f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient(hunting): picked mon.noname-c con 0x7f7bfc0a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bfc1360a0 0x7f7bfc12fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient(hunting): picked mon.noname-b con 0x7f7bfc1360a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1364f0 0x7f7bfc1d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient(hunting): picked mon.noname-a con 0x7f7bfc1364f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f7bfc140ff0 con 0x7f7bfc1364f0 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f7bfc140e70 con 0x7f7bfc1360a0 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient(hunting): start opening mon connection 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f7bfc141170 con 0x7f7bfc0a1990 2024-09-10T20:32:58.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient(hunting): _renew_subs 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7c01e34640 10 monclient(hunting): authenticate will time out at 1339.077976s 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfb7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc0a1990 msgr2=0x7f7bfc12f870 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfb7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc0a1990 0x7f7bfc12f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1364f0 0x7f7bfc1d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1364f0 0x7f7bfc1d04d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:55454/0 (socket says 172.21.15.153:55454) 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 1 -- 172.21.15.153:0/2374645728 learned_addr learned my addr 172.21.15.153:0/2374645728 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient(hunting): get_auth_request con 0x7f7bfc1364f0 auth_method 0 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:32:58.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfaffd640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bfc1360a0 0x7f7bfc12fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient(hunting): handle_auth_done global_id 34302 payload 824 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 1 -- 172.21.15.153:0/2374645728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc0a1990 msgr2=0x7f7bfc12f870 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc0a1990 0x7f7bfc12f870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 1 -- 172.21.15.153:0/2374645728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bfc1360a0 msgr2=0x7f7bfc12fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bfc1360a0 0x7f7bfc12fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient: _finish_hunting 0 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 1 monclient: found mon.noname-a 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfbfff640 1 -- 172.21.15.153:0/2374645728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bfc13ccd0 con 0x7f7bfc1364f0 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfaffd640 10 monclient: get_auth_request con 0x7f7bfc1360a0 auth_method 0 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.934+0000 7f7bfaffd640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bfc1360a0 0x7f7bfc12fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:32:58.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7bfbfff640 1 -- 172.21.15.153:0/2374645728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1364f0 msgr2=0x7f7bfc1d04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7bfbfff640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1364f0 0x7f7bfc1d04d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400c7b0 tx=0x7f7bf400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 client.?.objecter ms_handle_connect 0x7f7bfc1364f0 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 1 -- 172.21.15.153:0/2374645728 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf4017070 con 0x7f7bfc1364f0 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7bfbfff640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1364f0 0x7f7bfc1d04d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400c7b0 tx=0x7f7bf400cc70 comp rx=0 tx=0).queue_received will delay after 1.726e+09 on 0x7f7bf400ce60 mon_map magic: 0 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 monclient: dump: 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.954 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 monclient: _finish_auth 0 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 monclient: _check_auth_tickets 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 1 -- 172.21.15.153:0/2374645728 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bf4005070 con 0x7f7bfc1364f0 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 5 monclient: authenticate success, global_id 34302 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 10 monclient: _renew_subs 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bfc13cfb0 con 0x7f7bfc1364f0 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 10 client.34302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:32:58.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 10 monclient: _renew_subs 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7bf9ffb640 4 set_mon_vals no callback set 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bfc0a02a0 con 0x7f7bfc1364f0 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 1 librados: init done 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bfc1368d0 con 0x7f7bfc1364f0 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 1 -- 172.21.15.153:0/2374645728 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bf400ce60 con 0x7f7bfc1364f0 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 10 monclient: dump: 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:32:58.956 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 1 -- 172.21.15.153:0/2374645728 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f7bf4005460 con 0x7f7bfc1364f0 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7be3fff640 1 --2- 172.21.15.153:0/2374645728 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7be803ce50 0x7f7be803f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7bfb7fe640 1 --2- 172.21.15.153:0/2374645728 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7be803ce50 0x7f7be803f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:32:58.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.938+0000 7f7bfb7fe640 10 monclient: get_auth_request con 0x7f7be803ce50 auth_method 0 2024-09-10T20:32:58.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.942+0000 7f7bfb7fe640 1 --2- 172.21.15.153:0/2374645728 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7be803ce50 0x7f7be803f310 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0003990 tx=0x7f7bf0006cf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:32:58.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.942+0000 7f7be3fff640 10 client.34302.objecter ms_handle_connect 0x7f7be803ce50 2024-09-10T20:32:58.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.942+0000 7f7be3fff640 1 -- 172.21.15.153:0/2374645728 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(300..300 src has 1..300) ==== 5220+0+0 (secure 0 0 0) 0x7f7bf404f0a0 con 0x7f7bfc1364f0 2024-09-10T20:32:58.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.942+0000 7f7be3fff640 10 client.34302.objecter ms_dispatch 0x7f7bfc000ba0 osd_map(300..300 src has 1..300) 2024-09-10T20:32:58.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.942+0000 7f7be3fff640 3 client.34302.objecter handle_osd_map got epochs [300,300] > 0 2024-09-10T20:32:58.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.942+0000 7f7be3fff640 3 client.34302.objecter handle_osd_map decoding full epoch 300 2024-09-10T20:32:58.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:58.942+0000 7f7be3fff640 20 client.34302.objecter dump_active .. 0 homeless 2024-09-10T20:32:59.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.562+0000 7f7be3fff640 1 -- 172.21.15.153:0/2374645728 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f7bf404d8e0 con 0x7f7bfc1364f0 2024-09-10T20:32:59.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.562+0000 7f7be3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:32:59.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.562+0000 7f7be3fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:32:59.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.802+0000 7f7c01e34640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:32:59.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.802+0000 7f7c01e34640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:32:59.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.802+0000 7f7c01e34640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:32:59.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.802+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7bfc09fbc0 con 0x7f7bfc1364f0 2024-09-10T20:32:59.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.802+0000 7f7be3fff640 1 -- 172.21.15.153:0/2374645728 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f7bf4010300 con 0x7f7bfc1364f0 2024-09-10T20:32:59.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.802+0000 7f7be3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:32:59.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.802+0000 7f7be3fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:32:59.809 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:32:59.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.802+0000 7f7c01e34640 10 librados: watch_flush enter 2024-09-10T20:32:59.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 10 librados: watch_flush exit 2024-09-10T20:32:59.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 20 client.34302.objecter shutdown clearing up homeless session... 2024-09-10T20:32:59.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 10 client.34302.objecter successfully canceled tick 2024-09-10T20:32:59.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7be803ce50 msgr2=0x7f7be803f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:59.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 --2- 172.21.15.153:0/2374645728 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7be803ce50 0x7f7be803f310 secure :-1 s=READY pgs=3356 cs=0 l=1 rev1=1 crypto rx=0x7f7bf0003990 tx=0x7f7bf0006cf0 comp rx=0 tx=0).stop 2024-09-10T20:32:59.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 10 monclient: shutdown 2024-09-10T20:32:59.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1364f0 msgr2=0x7f7bfc1d04d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1364f0 0x7f7bfc1d04d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f7bf400c7b0 tx=0x7f7bf400cc70 comp rx=0 tx=0).stop 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 shutdown_connections 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f7bfc1364f0 0x7f7bfc1d04d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f7bfc1360a0 0x7f7bfc12fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 --2- 172.21.15.153:0/2374645728 >> v2:172.21.15.176:6812/3003473398 conn(0x7f7be803ce50 0x7f7be803f310 unknown :-1 s=CLOSED pgs=3356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 --2- 172.21.15.153:0/2374645728 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f7bfc0a1990 0x7f7bfc12f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 >> 172.21.15.153:0/2374645728 conn(0x7f7bfc0a71b0 msgr2=0x7f7bfc0a53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 shutdown_connections 2024-09-10T20:32:59.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 -- 172.21.15.153:0/2374645728 wait complete. 2024-09-10T20:32:59.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:32:59.806+0000 7f7c01e34640 1 librados: shutdown 2024-09-10T20:33:02.828 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:33:02.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient: get_monmap_and_config 2024-09-10T20:33:02.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient: build_initial_monmap 2024-09-10T20:33:02.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient: monmap: 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:33:02.958617+0000 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:33:02.958617+0000 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:33:02.960 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:02.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 1 Processor -- start 2024-09-10T20:33:02.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 1 -- start start 2024-09-10T20:33:02.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient: init 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a5ab0 0x7f8a200a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient(hunting): picked mon.noname-b con 0x7f8a200a5ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a200a63d0 addr v1:172.21.15.153:6789/0 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f8a201390d0 0x7f8a20140c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient(hunting): picked mon.noname-c con 0x7f8a201390d0 addr v2:172.21.15.176:3300/0 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a20070780 con 0x7f8a200a63d0 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f8a200a3010 con 0x7f8a201390d0 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:02.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8a200709c0 con 0x7f8a200a5ab0 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a27804640 10 monclient(hunting): authenticate will time out at 1343.098308s 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a255a0640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a5ab0 msgr2=0x7f8a200a5e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a255a0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a5ab0 0x7f8a200a5e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a24d9f640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8a200a63d0 0x7f8a201389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39254/0 (socket says 172.21.15.153:39254) 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a24d9f640 1 -- 172.21.15.153:0/2717566570 learned_addr learned my addr 172.21.15.153:0/2717566570 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 1 --2- 172.21.15.153:0/2717566570 >> v2:172.21.15.176:3300/0 conn(0x7f8a201390d0 0x7f8a20140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient(hunting): get_auth_request con 0x7f8a201390d0 auth_method 0 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:02.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient(hunting): handle_auth_done global_id 20152 payload 274 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 1 -- 172.21.15.153:0/2717566570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a5ab0 msgr2=0x7f8a200a5e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 1 --2- 172.21.15.153:0/2717566570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a5ab0 0x7f8a200a5e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 1 -- 172.21.15.153:0/2717566570 >> v1:172.21.15.153:6789/0 conn(0x7f8a200a63d0 legacy=0x7f8a201389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient: _finish_hunting 0 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 1 monclient: found mon.noname-c 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:33:02.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.954+0000 7f8a25da1640 1 -- 172.21.15.153:0/2717566570 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a200a26e0 con 0x7f8a201390d0 2024-09-10T20:33:02.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.958+0000 7f8a25da1640 1 -- 172.21.15.153:0/2717566570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a201390d0 msgr2=0x7f8a20140c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:02.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.958+0000 7f8a25da1640 1 --2- 172.21.15.153:0/2717566570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a201390d0 0x7f8a20140c90 secure :-1 s=READY pgs=5733 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009d90 tx=0x7f8a1801c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc686783832998ac server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:02.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 1 -- 172.21.15.153:0/2717566570 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180b2070 con 0x7f8a201390d0 2024-09-10T20:33:02.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:02.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:33:02.971 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 10 monclient: dump: 2024-09-10T20:33:02.971 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:02.971 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:02.971 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:02.971 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:02.972 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:02.972 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:02.972 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:02.972 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:02.972 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:02.972 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:02.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 10 monclient: _finish_auth 0 2024-09-10T20:33:02.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 10 monclient: _check_auth_tickets 2024-09-10T20:33:02.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 1 -- 172.21.15.153:0/2717566570 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a180a5990 con 0x7f8a201390d0 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 5 monclient: authenticate success, global_id 20152 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient: get_monmap_and_config success 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 1 -- 172.21.15.153:0/2717566570 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180a5d60 con 0x7f8a201390d0 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 4 set_mon_vals no callback set 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a17fff640 10 monclient: dump: 2024-09-10T20:33:02.973 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient: shutdown 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 -- 172.21.15.153:0/2717566570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a201390d0 msgr2=0x7f8a20140c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 --2- 172.21.15.153:0/2717566570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a201390d0 0x7f8a20140c90 secure :-1 s=READY pgs=5733 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009d90 tx=0x7f8a1801c850 comp rx=0 tx=0).stop 2024-09-10T20:33:02.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 -- 172.21.15.153:0/2717566570 shutdown_connections 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 --2- 172.21.15.153:0/2717566570 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a201390d0 0x7f8a20140c90 unknown :-1 s=CLOSED pgs=5733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 --2- 172.21.15.153:0/2717566570 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a5ab0 0x7f8a200a5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 -- 172.21.15.153:0/2717566570 >> 172.21.15.153:0/2717566570 conn(0x7f8a200aba00 msgr2=0x7f8a200a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 -- 172.21.15.153:0/2717566570 shutdown_connections 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 -- 172.21.15.153:0/2717566570 wait complete. 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient: build_initial_monmap 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient: monmap: 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:02.975 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 librados: starting msgr at 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 librados: starting objecter 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 Processor -- start 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 -- start start 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 librados: setting wanted keys 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 librados: calling monclient init 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient: init 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:02.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a200a5ab0 0x7f8a201d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient(hunting): picked mon.noname-b con 0x7f8a200a5ab0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a63d0 0x7f8a201379f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient(hunting): picked mon.noname-c con 0x7f8a200a63d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a201390d0 0x7f8a2013dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient(hunting): picked mon.noname-a con 0x7f8a201390d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8a200a3960 con 0x7f8a201390d0 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a200a3ae0 con 0x7f8a200a5ab0 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8a201439f0 con 0x7f8a200a63d0 2024-09-10T20:33:02.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a27804640 10 monclient(hunting): authenticate will time out at 1343.106579s 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a24d9f640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a63d0 msgr2=0x7f8a201379f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a24d9f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a63d0 0x7f8a201379f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a201390d0 0x7f8a2013dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a201390d0 0x7f8a2013dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38128/0 (socket says 172.21.15.153:38128) 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 1 -- 172.21.15.153:0/2957560622 learned_addr learned my addr 172.21.15.153:0/2957560622 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient(hunting): get_auth_request con 0x7f8a201390d0 auth_method 0 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:02.978 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a255a0640 1 --2- 172.21.15.153:0/2957560622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a200a5ab0 0x7f8a201d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a255a0640 10 monclient(hunting): get_auth_request con 0x7f8a200a5ab0 auth_method 0 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a255a0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a255a0640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a255a0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient(hunting): handle_auth_done global_id 34305 payload 824 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 1 -- 172.21.15.153:0/2957560622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a63d0 msgr2=0x7f8a201379f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 1 --2- 172.21.15.153:0/2957560622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a63d0 0x7f8a201379f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 1 -- 172.21.15.153:0/2957560622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a200a5ab0 msgr2=0x7f8a201d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 1 --2- 172.21.15.153:0/2957560622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a200a5ab0 0x7f8a201d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient: _finish_hunting 0 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 1 monclient: found mon.noname-a 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:02.979 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:33:02.980 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.962+0000 7f8a25da1640 1 -- 172.21.15.153:0/2957560622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a20122380 con 0x7f8a201390d0 2024-09-10T20:33:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a25da1640 1 -- 172.21.15.153:0/2957560622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a201390d0 msgr2=0x7f8a2013dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a25da1640 1 --2- 172.21.15.153:0/2957560622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a201390d0 0x7f8a2013dd50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009d90 tx=0x7f8a180195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 client.?.objecter ms_handle_connect 0x7f8a201390d0 2024-09-10T20:33:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:33:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 1 -- 172.21.15.153:0/2957560622 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180bd020 con 0x7f8a201390d0 2024-09-10T20:33:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:33:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 monclient: dump: 2024-09-10T20:33:02.982 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 monclient: _finish_auth 0 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:02.983 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 1 -- 172.21.15.153:0/2957560622 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a180b3040 con 0x7f8a201390d0 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 5 monclient: authenticate success, global_id 34305 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 10 monclient: _renew_subs 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a201402f0 con 0x7f8a201390d0 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 1 -- 172.21.15.153:0/2957560622 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a180b2070 con 0x7f8a201390d0 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 monclient: dump: 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:02.984 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a177fe640 4 set_mon_vals no callback set 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 10 client.34305.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 10 monclient: _renew_subs 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a20121ae0 con 0x7f8a201390d0 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 1 librados: init done 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 1 -- 172.21.15.153:0/2957560622 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f8a180c0020 con 0x7f8a201390d0 2024-09-10T20:33:02.985 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a200a5e90 con 0x7f8a201390d0 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 1 --2- 172.21.15.153:0/2957560622 >> v2:172.21.15.176:6812/3003473398 conn(0x7f8a0403ce50 0x7f8a0403f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 1 -- 172.21.15.153:0/2957560622 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(300..300 src has 1..300) ==== 5220+0+0 (secure 0 0 0) 0x7f8a180ea810 con 0x7f8a201390d0 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 10 client.34305.objecter ms_dispatch 0x7f8a20000ba0 osd_map(300..300 src has 1..300) 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 3 client.34305.objecter handle_osd_map got epochs [300,300] > 0 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 3 client.34305.objecter handle_osd_map decoding full epoch 300 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a15ffb640 20 client.34305.objecter dump_active .. 0 homeless 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a255a0640 1 --2- 172.21.15.153:0/2957560622 >> v2:172.21.15.176:6812/3003473398 conn(0x7f8a0403ce50 0x7f8a0403f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.966+0000 7f8a255a0640 10 monclient: get_auth_request con 0x7f8a0403ce50 auth_method 0 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.970+0000 7f8a255a0640 1 --2- 172.21.15.153:0/2957560622 >> v2:172.21.15.176:6812/3003473398 conn(0x7f8a0403ce50 0x7f8a0403f310 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f8a10006fd0 tx=0x7f8a10008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.970+0000 7f8a15ffb640 10 client.34305.objecter ms_handle_connect 0x7f8a0403ce50 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.970+0000 7f8a15ffb640 1 -- 172.21.15.153:0/2957560622 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f8a180c7070 con 0x7f8a201390d0 2024-09-10T20:33:02.986 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.970+0000 7f8a15ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:02.987 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:02.970+0000 7f8a15ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:33:03.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:33:03.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:33:03.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:03.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8a200a6be0 con 0x7f8a201390d0 2024-09-10T20:33:03.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a15ffb640 1 -- 172.21.15.153:0/2957560622 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f8a18016560 con 0x7f8a201390d0 2024-09-10T20:33:03.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a15ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:33:03.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a15ffb640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:33:03.200 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":7,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:33:03.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 10 librados: watch_flush enter 2024-09-10T20:33:03.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 10 librados: watch_flush exit 2024-09-10T20:33:03.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 20 client.34305.objecter shutdown clearing up homeless session... 2024-09-10T20:33:03.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 10 client.34305.objecter successfully canceled tick 2024-09-10T20:33:03.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 >> v2:172.21.15.176:6812/3003473398 conn(0x7f8a0403ce50 msgr2=0x7f8a0403f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 --2- 172.21.15.153:0/2957560622 >> v2:172.21.15.176:6812/3003473398 conn(0x7f8a0403ce50 0x7f8a0403f310 secure :-1 s=READY pgs=3357 cs=0 l=1 rev1=1 crypto rx=0x7f8a10006fd0 tx=0x7f8a10008040 comp rx=0 tx=0).stop 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 10 monclient: shutdown 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a201390d0 msgr2=0x7f8a2013dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 --2- 172.21.15.153:0/2957560622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a201390d0 0x7f8a2013dd50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009d90 tx=0x7f8a180195f0 comp rx=0 tx=0).stop 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 shutdown_connections 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 --2- 172.21.15.153:0/2957560622 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a201390d0 0x7f8a2013dd50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 --2- 172.21.15.153:0/2957560622 >> v2:172.21.15.176:6812/3003473398 conn(0x7f8a0403ce50 0x7f8a0403f310 unknown :-1 s=CLOSED pgs=3357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 --2- 172.21.15.153:0/2957560622 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a200a63d0 0x7f8a201379f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 --2- 172.21.15.153:0/2957560622 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a200a5ab0 0x7f8a201d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 >> 172.21.15.153:0/2957560622 conn(0x7f8a200aba00 msgr2=0x7f8a2009cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 shutdown_connections 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 -- 172.21.15.153:0/2957560622 wait complete. 2024-09-10T20:33:03.203 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:03.194+0000 7f8a27804640 1 librados: shutdown 2024-09-10T20:33:06.217 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:33:06.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient: get_monmap_and_config 2024-09-10T20:33:06.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient: build_initial_monmap 2024-09-10T20:33:06.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient: monmap: 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:33:06.346551+0000 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:33:06.346551+0000 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:33:06.348 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:06.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 1 Processor -- start 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 1 -- start start 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient: init 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 0x7fd2ac1368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient(hunting): picked mon.noname-b con 0x7fd2ac1364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd2ac1360a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fd2ac0a1990 0x7fd2ac09d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient(hunting): picked mon.noname-c con 0x7fd2ac0a1990 addr v2:172.21.15.176:3300/0 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2ac0a27a0 con 0x7fd2ac1360a0 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:06.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fd2ac0a2a80 con 0x7fd2ac0a1990 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ac0a2d60 con 0x7fd2ac1364f0 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2b1b03640 10 monclient(hunting): authenticate will time out at 1346.486184s 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2ab7fe640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 msgr2=0x7fd2ac1368d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2ab7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 0x7fd2ac1368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2aaffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd2ac1360a0 0x7fd2ac09ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:39258/0 (socket says 172.21.15.153:39258) 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2aaffd640 1 -- 172.21.15.153:0/754516784 learned_addr learned my addr 172.21.15.153:0/754516784 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 1 --2- 172.21.15.153:0/754516784 >> v2:172.21.15.176:3300/0 conn(0x7fd2ac0a1990 0x7fd2ac09d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient(hunting): get_auth_request con 0x7fd2ac0a1990 auth_method 0 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:06.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient(hunting): handle_auth_done global_id 20158 payload 274 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 1 -- 172.21.15.153:0/754516784 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 msgr2=0x7fd2ac1368d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 1 --2- 172.21.15.153:0/754516784 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 0x7fd2ac1368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 1 -- 172.21.15.153:0/754516784 >> v1:172.21.15.153:6789/0 conn(0x7fd2ac1360a0 legacy=0x7fd2ac09ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient: _finish_hunting 0 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 1 monclient: found mon.noname-c 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:33:06.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.342+0000 7fd2abfff640 1 -- 172.21.15.153:0/754516784 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ac136e10 con 0x7fd2ac0a1990 2024-09-10T20:33:06.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2abfff640 1 -- 172.21.15.153:0/754516784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac0a1990 msgr2=0x7fd2ac09d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:06.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2abfff640 1 --2- 172.21.15.153:0/754516784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac0a1990 0x7fd2ac09d520 secure :-1 s=READY pgs=5735 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009d90 tx=0x7fd2a401c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cdb142932a51daf8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 1 -- 172.21.15.153:0/754516784 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40b2070 con 0x7fd2ac0a1990 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 10 monclient: dump: 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:06.356 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 10 monclient: _finish_auth 0 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 1 -- 172.21.15.153:0/754516784 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd2ac09dc70 con 0x7fd2ac0a1990 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 1 -- 172.21.15.153:0/754516784 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2a40a5bb0 con 0x7fd2ac0a1990 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 5 monclient: authenticate success, global_id 20158 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient: get_monmap_and_config success 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 4 set_mon_vals no callback set 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:33:06.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aa7fc640 10 monclient: dump: 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient: shutdown 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 -- 172.21.15.153:0/754516784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac0a1990 msgr2=0x7fd2ac09d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:06.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 --2- 172.21.15.153:0/754516784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac0a1990 0x7fd2ac09d520 secure :-1 s=READY pgs=5735 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4009d90 tx=0x7fd2a401c850 comp rx=0 tx=0).stop 2024-09-10T20:33:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 -- 172.21.15.153:0/754516784 shutdown_connections 2024-09-10T20:33:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 --2- 172.21.15.153:0/754516784 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac0a1990 0x7fd2ac09d520 unknown :-1 s=CLOSED pgs=5735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 --2- 172.21.15.153:0/754516784 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 0x7fd2ac1368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 -- 172.21.15.153:0/754516784 >> 172.21.15.153:0/754516784 conn(0x7fd2ac0a71b0 msgr2=0x7fd2ac0a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 -- 172.21.15.153:0/754516784 shutdown_connections 2024-09-10T20:33:06.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 -- 172.21.15.153:0/754516784 wait complete. 2024-09-10T20:33:06.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient: build_initial_monmap 2024-09-10T20:33:06.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient: monmap: 2024-09-10T20:33:06.363 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:06.363 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 librados: starting msgr at 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 librados: starting objecter 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 Processor -- start 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 -- start start 2024-09-10T20:33:06.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 librados: setting wanted keys 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 librados: calling monclient init 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient: init 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0a1990 0x7fd2ac0730f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient(hunting): picked mon.noname-a con 0x7fd2ac0a1990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac1360a0 0x7fd2ac073630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient(hunting): picked mon.noname-b con 0x7fd2ac1360a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 0x7fd2ac073b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient(hunting): picked mon.noname-c con 0x7fd2ac1364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac09e6c0 con 0x7fd2ac0a1990 2024-09-10T20:33:06.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd2ac09e540 con 0x7fd2ac1360a0 2024-09-10T20:33:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd2ac09e840 con 0x7fd2ac1364f0 2024-09-10T20:33:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2b1b03640 10 monclient(hunting): authenticate will time out at 1346.495068s 2024-09-10T20:33:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2abfff640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 msgr2=0x7fd2ac073b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2abfff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 0x7fd2ac073b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0a1990 0x7fd2ac0730f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:06.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0a1990 0x7fd2ac0730f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38132/0 (socket says 172.21.15.153:38132) 2024-09-10T20:33:06.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 1 -- 172.21.15.153:0/2881494554 learned_addr learned my addr 172.21.15.153:0/2881494554 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:33:06.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient(hunting): get_auth_request con 0x7fd2ac0a1990 auth_method 0 2024-09-10T20:33:06.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:06.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:06.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:06.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:06.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient(hunting): handle_auth_done global_id 34308 payload 824 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 1 -- 172.21.15.153:0/2881494554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 msgr2=0x7fd2ac073b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 1 --2- 172.21.15.153:0/2881494554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 0x7fd2ac073b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2aaffd640 1 --2- 172.21.15.153:0/2881494554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac1360a0 0x7fd2ac073630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 1 -- 172.21.15.153:0/2881494554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac1360a0 msgr2=0x7fd2ac073630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 1 --2- 172.21.15.153:0/2881494554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac1360a0 0x7fd2ac073630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 1 monclient: found mon.noname-a 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.350+0000 7fd2ab7fe640 1 -- 172.21.15.153:0/2881494554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2ac139a50 con 0x7fd2ac0a1990 2024-09-10T20:33:06.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2ab7fe640 1 -- 172.21.15.153:0/2881494554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0a1990 msgr2=0x7fd2ac0730f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2ab7fe640 1 --2- 172.21.15.153:0/2881494554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0a1990 0x7fd2ac0730f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00b550 tx=0x7fd29c00ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 client.?.objecter ms_handle_connect 0x7fd2ac0a1990 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 1 -- 172.21.15.153:0/2881494554 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0050f0 con 0x7fd2ac0a1990 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 monclient: dump: 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:06.369 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 monclient: _finish_auth 0 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 monclient: _check_auth_tickets 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 5 monclient: authenticate success, global_id 34308 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 1 -- 172.21.15.153:0/2881494554 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd29c015070 con 0x7fd2ac0a1990 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 10 monclient: _renew_subs 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2ac1372f0 con 0x7fd2ac0a1990 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 1 -- 172.21.15.153:0/2881494554 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29c0054d0 con 0x7fd2ac0a1990 2024-09-10T20:33:06.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 monclient: dump: 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 10 client.34308.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 10 monclient: _renew_subs 2024-09-10T20:33:06.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2ac09e0a0 con 0x7fd2ac0a1990 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2a9ffb640 4 set_mon_vals no callback set 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 1 librados: init done 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2ac1368d0 con 0x7fd2ac0a1990 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 1 -- 172.21.15.153:0/2881494554 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fd29c005670 con 0x7fd2ac0a1990 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 1 --2- 172.21.15.153:0/2881494554 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd29803ce50 0x7fd29803f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 1 -- 172.21.15.153:0/2881494554 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(300..300 src has 1..300) ==== 5220+0+0 (secure 0 0 0) 0x7fd29c04f890 con 0x7fd2ac0a1990 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 10 client.34308.objecter ms_dispatch 0x7fd2ac000ba0 osd_map(300..300 src has 1..300) 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 3 client.34308.objecter handle_osd_map got epochs [300,300] > 0 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 3 client.34308.objecter handle_osd_map decoding full epoch 300 2024-09-10T20:33:06.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd293fff640 20 client.34308.objecter dump_active .. 0 homeless 2024-09-10T20:33:06.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2aaffd640 1 --2- 172.21.15.153:0/2881494554 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd29803ce50 0x7fd29803f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:06.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.354+0000 7fd2aaffd640 10 monclient: get_auth_request con 0x7fd29803ce50 auth_method 0 2024-09-10T20:33:06.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.358+0000 7fd293fff640 10 client.34308.objecter ms_handle_connect 0x7fd29803ce50 2024-09-10T20:33:06.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.358+0000 7fd2aaffd640 1 --2- 172.21.15.153:0/2881494554 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd29803ce50 0x7fd29803f310 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac13d680 tx=0x7fd2a0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:06.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.358+0000 7fd293fff640 1 -- 172.21.15.153:0/2881494554 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fd2ac139a50 con 0x7fd2ac0a1990 2024-09-10T20:33:06.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.358+0000 7fd293fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:06.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.358+0000 7fd293fff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:33:06.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:33:06.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:33:06.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:06.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd2ac06a660 con 0x7fd2ac0a1990 2024-09-10T20:33:06.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd293fff640 1 -- 172.21.15.153:0/2881494554 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1569 (secure 0 0 0) 0x7fd29c00ee20 con 0x7fd2ac0a1990 2024-09-10T20:33:06.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd293fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:33:06.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd293fff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:33:06.600 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:33:06.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 10 librados: watch_flush enter 2024-09-10T20:33:06.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 10 librados: watch_flush exit 2024-09-10T20:33:06.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 20 client.34308.objecter shutdown clearing up homeless session... 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 10 client.34308.objecter successfully canceled tick 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd29803ce50 msgr2=0x7fd29803f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 --2- 172.21.15.153:0/2881494554 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd29803ce50 0x7fd29803f310 secure :-1 s=READY pgs=3358 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac13d680 tx=0x7fd2a0009040 comp rx=0 tx=0).stop 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 10 monclient: shutdown 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0a1990 msgr2=0x7fd2ac0730f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 --2- 172.21.15.153:0/2881494554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0a1990 0x7fd2ac0730f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd29c00b550 tx=0x7fd29c00ba10 comp rx=0 tx=0).stop 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 shutdown_connections 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 --2- 172.21.15.153:0/2881494554 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd29803ce50 0x7fd29803f310 unknown :-1 s=CLOSED pgs=3358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 --2- 172.21.15.153:0/2881494554 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd2ac1364f0 0x7fd2ac073b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 --2- 172.21.15.153:0/2881494554 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd2ac1360a0 0x7fd2ac073630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 --2- 172.21.15.153:0/2881494554 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd2ac0a1990 0x7fd2ac0730f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 >> 172.21.15.153:0/2881494554 conn(0x7fd2ac0a71b0 msgr2=0x7fd2ac0a6520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:06.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 shutdown_connections 2024-09-10T20:33:06.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 -- 172.21.15.153:0/2881494554 wait complete. 2024-09-10T20:33:06.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:06.594+0000 7fd2b1b03640 1 librados: shutdown 2024-09-10T20:33:09.620 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient: get_monmap_and_config 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient: build_initial_monmap 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient: monmap: 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:33:09.743851+0000 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:33:09.743851+0000 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:33:09.745 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:33:09.746 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:33:09.746 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:33:09.746 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 1 Processor -- start 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 1 -- start start 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient: init 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient: _add_conns ranks=[3,4,1,2,0] 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient(hunting): picked mon.noname-a con 0x7fda681364f0 addr v2:172.21.15.153:3300/0 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient(hunting): picked mon.noname-c con 0x7fda681360a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:33:09.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fda680a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda680a27a0 con 0x7fda680a1990 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fda680a2a80 con 0x7fda681364f0 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fda680a2d60 con 0x7fda681360a0 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda70235640 10 monclient(hunting): authenticate will time out at 1349.883610s 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:38142/0 (socket says 172.21.15.153:38142) 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 learned_addr learned my addr 172.21.15.153:0/779788687 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request con 0x7fda681364f0 auth_method 0 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:09.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 msgr2=0x7fda681368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 msgr2=0x7fda681368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.738+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request con 0x7fda681360a0 auth_method 0 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6d7d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6d7d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 msgr2=0x7fda6809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 msgr2=0x7fda6809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:33:09.750 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:09.751 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.742+0000 7fda6e7d2640 1 -- 172.21.15.153:0/779788687 >> v1:172.21.15.176:6789/0 conn(0x7fda680a1990 legacy=0x7fda6809d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:09.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request con 0x7fda681364f0 auth_method 0 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6dfd1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6dfd1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request con 0x7fda681360a0 auth_method 0 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6d7d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6d7d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 msgr2=0x7fda681368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 msgr2=0x7fda681368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:33:09.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:33:09.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 msgr2=0x7fda6809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:33:09.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 msgr2=0x7fda6809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:33:09.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:33:09.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:09.946+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request con 0x7fda681364f0 auth_method 0 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6dfd1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6dfd1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 msgr2=0x7fda681368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 msgr2=0x7fda681368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request con 0x7fda681360a0 auth_method 0 2024-09-10T20:33:10.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:10.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6d7d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:10.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6d7d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:33:10.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:33:10.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 msgr2=0x7fda6809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:33:10.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 msgr2=0x7fda6809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:33:10.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:33:10.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:10.350+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:33:11.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:11.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request con 0x7fda681364f0 auth_method 0 2024-09-10T20:33:11.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:11.157 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6dfd1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6dfd1640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 msgr2=0x7fda681368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 msgr2=0x7fda681368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request con 0x7fda681360a0 auth_method 0 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6d7d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6d7d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 msgr2=0x7fda6809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 msgr2=0x7fda6809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:33:11.158 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:11.154+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-09-10T20:33:12.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 10 monclient(hunting): tick 2024-09-10T20:33:12.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-10T20:33:12.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 1 monclient(hunting): continuing hunt 2024-09-10T20:33:12.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-10T20:33:12.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 msgr2=0x7fda6809ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:33:12.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.176:3300/0 conn(0x7fda681360a0 0x7fda6809ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:12.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 1 -- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 msgr2=0x7fda681368d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:33:12.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 1 --2- 172.21.15.153:0/779788687 >> v2:172.21.15.153:3300/0 conn(0x7fda681364f0 0x7fda681368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:12.747 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 1 -- 172.21.15.153:0/779788687 >> v1:172.21.15.176:6789/0 conn(0x7fda680a1990 legacy=0x7fda6809d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 10 monclient: _add_conns ranks=[0,2] 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda4c003740 addr v1:172.21.15.153:6789/0 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0073f0 0x7fda4c009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fda4c0073f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 1 -- 172.21.15.153:0/779788687 --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda68136e10 con 0x7fda4c003740 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 1 -- 172.21.15.153:0/779788687 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda680a2a80 con 0x7fda4c0073f0 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda5f7fe640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0073f0 msgr2=0x7fda4c009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:12.748 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0073f0 0x7fda4c009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:12.749 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.742+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> v1:172.21.15.153:6789/0 conn(0x7fda4c003740 legacy=0x7fda4c005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.946+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0073f0 msgr2=0x7fda4c009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:12.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:12.946+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0073f0 0x7fda4c009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:33:13.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.350+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0073f0 msgr2=0x7fda4c009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:13.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.350+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0073f0 0x7fda4c009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:33:13.801 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.794+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106550199 0 0) 0x7fda68136e10 con 0x7fda4c003740 2024-09-10T20:33:13.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.794+0000 7fda6cfcf640 10 monclient(hunting): _negotiate 2024-09-10T20:33:13.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.794+0000 7fda6cfcf640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:13.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.794+0000 7fda6cfcf640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:13.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.794+0000 7fda6cfcf640 10 monclient(hunting): my global_id is 34311 2024-09-10T20:33:13.802 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.794+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fda54003530 con 0x7fda4c003740 2024-09-10T20:33:13.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3143461878 0 0) 0x7fda54003530 con 0x7fda4c003740 2024-09-10T20:33:13.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0073f0 msgr2=0x7fda4c009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:33:13.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda4c0073f0 0x7fda4c009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:13.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient: _finish_hunting 0 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 monclient: found mon.noname-a-legacy 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda4c00b070 con 0x7fda4c003740 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient: _finish_auth 0 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient: _check_auth_tickets 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3765011595 0 0) 0x7fda6400f040 con 0x7fda4c003740 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda70235640 5 monclient: authenticate success, global_id 34311 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda70235640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T20:33:13.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient: dump: 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:13.807 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 >> v1:172.21.15.153:6789/0 conn(0x7fda4c003740 legacy=0x7fda4c005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6e7d2640 1 -- 172.21.15.153:0/779788687 reap_dead start 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda540039c0 0x7fda54003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient(hunting): picked mon.c con 0x7fda540039c0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54004810 0x7fda54004c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient(hunting): picked mon.b con 0x7fda54004810 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda54005380 0x7fda5400b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient(hunting): picked mon.a con 0x7fda54005380 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:33:13.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda680a2a80 con 0x7fda54005380 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda5400c2d0 con 0x7fda54004810 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda5400c4d0 con 0x7fda540039c0 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6cfcf640 10 monclient(hunting): _finish_auth 0 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6dfd1640 1 -- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda540039c0 msgr2=0x7fda54003dc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda70235640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda540039c0 0x7fda54003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6e7d2640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda54005380 0x7fda5400b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:13.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54004810 0x7fda54004c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request con 0x7fda54004810 auth_method 0 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6e7d2640 10 monclient(hunting): get_auth_request con 0x7fda54005380 auth_method 0 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6e7d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6e7d2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6e7d2640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6e7d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:13.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6e7d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6e7d2640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient(hunting): handle_auth_done global_id 34311 payload 293 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda540039c0 msgr2=0x7fda54003dc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda540039c0 0x7fda54003dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda54005380 msgr2=0x7fda5400b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda54005380 0x7fda5400b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient: _finish_hunting 0 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 1 monclient: found mon.b 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient: _send_mon_message to mon.b at v2:172.21.15.176:3300/0 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda54003530 con 0x7fda54004810 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient: _finish_auth 0 2024-09-10T20:33:13.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 10 monclient: _check_auth_tickets 2024-09-10T20:33:13.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.798+0000 7fda6d7d0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:13.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6d7d0640 1 -- 172.21.15.153:0/779788687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54004810 msgr2=0x7fda54004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:13.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54004810 0x7fda54004c90 secure :-1 s=READY pgs=5738 cs=0 l=1 rev1=1 crypto rx=0x7fda5800eae0 tx=0x7fda5801d3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8811f456053af9fa server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:13.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda580b3070 con 0x7fda54004810 2024-09-10T20:33:13.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:13.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 10 monclient: dump: 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda70235640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:33:13.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda580ae410 con 0x7fda54004810 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda70235640 10 monclient: get_monmap_and_config success 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda70235640 4 set_mon_vals no callback set 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 1 -- 172.21.15.153:0/779788687 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda5800f040 con 0x7fda54004810 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.802+0000 7fda6cfcf640 10 monclient: dump: 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:13.815 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 10 monclient: shutdown 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 -- 172.21.15.153:0/779788687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54004810 msgr2=0x7fda54004c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54004810 0x7fda54004c90 secure :-1 s=READY pgs=5738 cs=0 l=1 rev1=1 crypto rx=0x7fda5800eae0 tx=0x7fda5801d3d0 comp rx=0 tx=0).stop 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 -- 172.21.15.153:0/779788687 shutdown_connections 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda54005380 0x7fda5400b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54004810 0x7fda54004c90 unknown :-1 s=CLOSED pgs=5738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:13.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 --2- 172.21.15.153:0/779788687 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda540039c0 0x7fda54003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 -- 172.21.15.153:0/779788687 >> 172.21.15.153:0/779788687 conn(0x7fda680a71b0 msgr2=0x7fda680a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 -- 172.21.15.153:0/779788687 shutdown_connections 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 -- 172.21.15.153:0/779788687 wait complete. 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 10 monclient: build_initial_monmap 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 10 monclient: monmap: 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:33:13.817 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:33:13.818 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:33:13.818 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:13.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 librados: starting msgr at 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 librados: starting objecter 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 Processor -- start 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 -- start start 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 librados: setting wanted keys 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 1 librados: calling monclient init 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.806+0000 7fda70235640 10 monclient: init 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda540039c0 0x7fda681cd310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient(hunting): picked mon.noname-a con 0x7fda540039c0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda54004810 0x7fda6812f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:13.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient(hunting): picked mon.noname-c con 0x7fda54004810 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54005380 0x7fda681bde80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient(hunting): picked mon.noname-b con 0x7fda54005380 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda540039c0 0x7fda681cd310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda681368d0 con 0x7fda540039c0 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6d7d0640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda54004810 msgr2=0x7fda6812f7e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda540039c0 0x7fda681cd310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39798/0 (socket says 172.21.15.153:39798) 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 1 -- 172.21.15.153:0/2757860831 learned_addr learned my addr 172.21.15.153:0/2757860831 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/2757860831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda54004810 0x7fda6812f7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda680a08a0 con 0x7fda54005380 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda68136a50 con 0x7fda54004810 2024-09-10T20:33:13.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda70235640 10 monclient(hunting): authenticate will time out at 1353.954207s 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request con 0x7fda540039c0 auth_method 0 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6e7d2640 1 --2- 172.21.15.153:0/2757860831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54005380 0x7fda681bde80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6e7d2640 10 monclient(hunting): get_auth_request con 0x7fda54005380 auth_method 0 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6e7d2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6e7d2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6e7d2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:13.826 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6e7d2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6e7d2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6e7d2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient(hunting): handle_auth_done global_id 34314 payload 824 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 1 -- 172.21.15.153:0/2757860831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda54004810 msgr2=0x7fda6812f7e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/2757860831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda54004810 0x7fda6812f7e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 1 -- 172.21.15.153:0/2757860831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54005380 msgr2=0x7fda681bde80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/2757860831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54005380 0x7fda681bde80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient: _finish_hunting 0 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 1 monclient: found mon.noname-a 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.810+0000 7fda6dfd1640 1 -- 172.21.15.153:0/2757860831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda68133c70 con 0x7fda540039c0 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda6dfd1640 1 -- 172.21.15.153:0/2757860831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda540039c0 msgr2=0x7fda681cd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:13.827 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda6dfd1640 1 --2- 172.21.15.153:0/2757860831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda540039c0 0x7fda681cd310 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fda64015910 tx=0x7fda64015dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 client.?.objecter ms_handle_connect 0x7fda540039c0 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 1 -- 172.21.15.153:0/2757860831 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda64013c10 con 0x7fda540039c0 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 monclient: dump: 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:13.828 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 monclient: _finish_auth 0 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 5 monclient: authenticate success, global_id 34314 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 10 monclient: _renew_subs 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 1 -- 172.21.15.153:0/2757860831 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda64013db0 con 0x7fda540039c0 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda681bf850 con 0x7fda540039c0 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 1 -- 172.21.15.153:0/2757860831 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda64004270 con 0x7fda540039c0 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:13.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 monclient: dump: 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5ffff640 4 set_mon_vals no callback set 2024-09-10T20:33:13.830 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 10 client.34314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:33:13.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 10 monclient: _renew_subs 2024-09-10T20:33:13.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:13.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda681c03b0 con 0x7fda540039c0 2024-09-10T20:33:13.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 1 librados: init done 2024-09-10T20:33:13.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:13.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:13.831 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 1 -- 172.21.15.153:0/2757860831 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fda64003620 con 0x7fda540039c0 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda680a2d60 con 0x7fda540039c0 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 1 --2- 172.21.15.153:0/2757860831 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda4c03d850 0x7fda4c03fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 1 -- 172.21.15.153:0/2757860831 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(301..301 src has 1..301) ==== 4803+0+0 (secure 0 0 0) 0x7fda640530b0 con 0x7fda540039c0 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 10 client.34314.objecter ms_dispatch 0x7fda68000ba0 osd_map(301..301 src has 1..301) 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 3 client.34314.objecter handle_osd_map got epochs [301,301] > 0 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 3 client.34314.objecter handle_osd_map decoding full epoch 301 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda5e7fc640 20 client.34314.objecter dump_active .. 0 homeless 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.814+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/2757860831 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda4c03d850 0x7fda4c03fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:13.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.818+0000 7fda6d7d0640 10 monclient: get_auth_request con 0x7fda4c03d850 auth_method 0 2024-09-10T20:33:13.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.818+0000 7fda5e7fc640 10 client.34314.objecter ms_handle_connect 0x7fda4c03d850 2024-09-10T20:33:13.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.818+0000 7fda6d7d0640 1 --2- 172.21.15.153:0/2757860831 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda4c03d850 0x7fda4c03fd10 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fda58000ee0 tx=0x7fda58010040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:13.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.822+0000 7fda5e7fc640 1 -- 172.21.15.153:0/2757860831 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fda64051a90 con 0x7fda540039c0 2024-09-10T20:33:13.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.822+0000 7fda5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:13.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:13.822+0000 7fda5e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:33:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.058+0000 7fda70235640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:33:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.058+0000 7fda70235640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:33:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.058+0000 7fda70235640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:14.066 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.058+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fda681ce190 con 0x7fda540039c0 2024-09-10T20:33:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda5e7fc640 1 -- 172.21.15.153:0/2757860831 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fda64018470 con 0x7fda540039c0 2024-09-10T20:33:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:33:14.067 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda5e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:33:14.067 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:33:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 10 librados: watch_flush enter 2024-09-10T20:33:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 10 librados: watch_flush exit 2024-09-10T20:33:14.069 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 20 client.34314.objecter shutdown clearing up homeless session... 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 10 client.34314.objecter successfully canceled tick 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda4c03d850 msgr2=0x7fda4c03fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 --2- 172.21.15.153:0/2757860831 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda4c03d850 0x7fda4c03fd10 secure :-1 s=READY pgs=3359 cs=0 l=1 rev1=1 crypto rx=0x7fda58000ee0 tx=0x7fda58010040 comp rx=0 tx=0).stop 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 10 monclient: shutdown 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda540039c0 msgr2=0x7fda681cd310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 --2- 172.21.15.153:0/2757860831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda540039c0 0x7fda681cd310 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fda64015910 tx=0x7fda64015dd0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 shutdown_connections 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 --2- 172.21.15.153:0/2757860831 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda54005380 0x7fda681bde80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 --2- 172.21.15.153:0/2757860831 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda4c03d850 0x7fda4c03fd10 unknown :-1 s=CLOSED pgs=3359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 --2- 172.21.15.153:0/2757860831 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda54004810 0x7fda6812f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 --2- 172.21.15.153:0/2757860831 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda540039c0 0x7fda681cd310 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 >> 172.21.15.153:0/2757860831 conn(0x7fda680a71b0 msgr2=0x7fda68132080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 shutdown_connections 2024-09-10T20:33:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 -- 172.21.15.153:0/2757860831 wait complete. 2024-09-10T20:33:14.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.062+0000 7fda70235640 1 librados: shutdown 2024-09-10T20:33:14.085 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-10T20:33:14.085 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-10T20:33:14.085 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:33:14.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.186+0000 7f55d5037640 10 monclient: get_monmap_and_config 2024-09-10T20:33:14.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.186+0000 7f55d5037640 10 monclient: build_initial_monmap 2024-09-10T20:33:14.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.186+0000 7f55d5037640 10 monclient: monmap: 2024-09-10T20:33:14.194 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:14.194 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:33:14.193698+0000 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:33:14.193698+0000 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:33:14.195 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.197 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 1 Processor -- start 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 1 -- start start 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient: init 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient: _add_conns ranks=[2,0,4,3,1] 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a41a0 0x7f55d00a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient(hunting): picked mon.noname-b con 0x7f55d00a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f55d00a4580 addr v1:172.21.15.153:6789/0 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f55d00a49a0 0x7f55d00a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient(hunting): picked mon.noname-c con 0x7f55d00a49a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f55d00a6a20 con 0x7f55d00a4580 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f55d00a2570 con 0x7f55d00a49a0 2024-09-10T20:33:14.198 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f55d0070400 con 0x7f55d00a41a0 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55d5037640 10 monclient(hunting): authenticate will time out at 1354.333302s 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55ced9d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a41a0 msgr2=0x7f55d00a3740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55ced9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a41a0 0x7f55d00a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55ce59c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f55d00a4580 0x7f55d00a59d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:45110/0 (socket says 172.21.15.153:45110) 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55ce59c640 1 -- 172.21.15.153:0/1388899722 learned_addr learned my addr 172.21.15.153:0/1388899722 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 1 --2- 172.21.15.153:0/1388899722 >> v2:172.21.15.176:3300/0 conn(0x7f55d00a49a0 0x7f55d00a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient(hunting): get_auth_request con 0x7f55d00a49a0 auth_method 0 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:14.199 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient(hunting): handle_auth_done global_id 20164 payload 274 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 1 -- 172.21.15.153:0/1388899722 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a41a0 msgr2=0x7f55d00a3740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 1 --2- 172.21.15.153:0/1388899722 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a41a0 0x7f55d00a3740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 1 -- 172.21.15.153:0/1388899722 >> v1:172.21.15.153:6789/0 conn(0x7f55d00a4580 legacy=0x7f55d00a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient: _finish_hunting 0 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 1 monclient: found mon.noname-c 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:33:14.200 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.190+0000 7f55cf59e640 1 -- 172.21.15.153:0/1388899722 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d00a3c80 con 0x7f55d00a49a0 2024-09-10T20:33:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.194+0000 7f55cf59e640 1 -- 172.21.15.153:0/1388899722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d00a49a0 msgr2=0x7f55d00a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:14.202 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.194+0000 7f55cf59e640 1 --2- 172.21.15.153:0/1388899722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d00a49a0 0x7f55d00a60e0 secure :-1 s=READY pgs=5739 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009d90 tx=0x7f55c001c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54381cfc8f0ef379 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 1 -- 172.21.15.153:0/1388899722 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00a86d0 con 0x7f55d00a49a0 2024-09-10T20:33:14.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:33:14.206 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 10 monclient: dump: 2024-09-10T20:33:14.206 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.206 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 10 monclient: _finish_auth 0 2024-09-10T20:33:14.207 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 10 monclient: _check_auth_tickets 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 1 -- 172.21.15.153:0/1388899722 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c00a8870 con 0x7f55d00a49a0 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 5 monclient: authenticate success, global_id 20164 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient: get_monmap_and_config success 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 4 set_mon_vals no callback set 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 1 -- 172.21.15.153:0/1388899722 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00a8c40 con 0x7f55d00a49a0 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.208 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cdd9b640 10 monclient: dump: 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient: shutdown 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:14.209 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 -- 172.21.15.153:0/1388899722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d00a49a0 msgr2=0x7f55d00a60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 --2- 172.21.15.153:0/1388899722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d00a49a0 0x7f55d00a60e0 secure :-1 s=READY pgs=5739 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009d90 tx=0x7f55c001c850 comp rx=0 tx=0).stop 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 -- 172.21.15.153:0/1388899722 shutdown_connections 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 --2- 172.21.15.153:0/1388899722 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d00a49a0 0x7f55d00a60e0 unknown :-1 s=CLOSED pgs=5739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 --2- 172.21.15.153:0/1388899722 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a41a0 0x7f55d00a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 -- 172.21.15.153:0/1388899722 >> 172.21.15.153:0/1388899722 conn(0x7f55d00ab7e0 msgr2=0x7f55d00a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 -- 172.21.15.153:0/1388899722 shutdown_connections 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 -- 172.21.15.153:0/1388899722 wait complete. 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient: build_initial_monmap 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient: monmap: 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:33:14.210 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 librados: starting msgr at 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 librados: starting objecter 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 Processor -- start 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 -- start start 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 librados: setting wanted keys 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 librados: calling monclient init 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient: init 2024-09-10T20:33:14.211 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d00a41a0 0x7f55d01259d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient(hunting): picked mon.noname-b con 0x7f55d00a41a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a4580 0x7f55d013c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient(hunting): picked mon.noname-c con 0x7f55d00a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d00a49a0 0x7f55d01d05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient(hunting): picked mon.noname-a con 0x7f55d00a49a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55d0143210 con 0x7f55d00a49a0 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f55d00a6830 con 0x7f55d00a41a0 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f55d0143390 con 0x7f55d00a4580 2024-09-10T20:33:14.212 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55d5037640 10 monclient(hunting): authenticate will time out at 1354.342151s 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55ce59c640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a4580 msgr2=0x7f55d013c840 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55ce59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a4580 0x7f55d013c840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d00a49a0 0x7f55d01d05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d00a49a0 0x7f55d01d05c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39804/0 (socket says 172.21.15.153:39804) 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 1 -- 172.21.15.153:0/1298792648 learned_addr learned my addr 172.21.15.153:0/1298792648 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient(hunting): get_auth_request con 0x7f55d00a49a0 auth_method 0 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:14.213 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55ced9d640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d00a41a0 0x7f55d01259d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55ced9d640 10 monclient(hunting): get_auth_request con 0x7f55d00a41a0 auth_method 0 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient(hunting): handle_auth_done global_id 34317 payload 824 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a4580 msgr2=0x7f55d013c840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f55d00a4580 0x7f55d013c840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d00a41a0 msgr2=0x7f55d01259d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f55d00a41a0 0x7f55d01259d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient: _finish_hunting 0 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 1 monclient: found mon.noname-a 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:14.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:33:14.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.198+0000 7f55cf59e640 1 -- 172.21.15.153:0/1298792648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d01271d0 con 0x7f55d00a49a0 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55cf59e640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d00a49a0 msgr2=0x7f55d01d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55cf59e640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d00a49a0 0x7f55d01d05c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f55c001c680 tx=0x7f55c00195f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 client.?.objecter ms_handle_connect 0x7f55d00a49a0 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00b0810 con 0x7f55d00a49a0 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 monclient: dump: 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.219 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 monclient: _finish_auth 0 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 monclient: _check_auth_tickets 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f55c00bf070 con 0x7f55d00a49a0 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 5 monclient: authenticate success, global_id 34317 2024-09-10T20:33:14.220 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 10 monclient: _renew_subs 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00b0bb0 con 0x7f55d00a49a0 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d0124e70 con 0x7f55d00a49a0 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 monclient: dump: 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.221 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55cd59a640 4 set_mon_vals no callback set 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 10 client.34317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 10 monclient: _renew_subs 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d0126c10 con 0x7f55d00a49a0 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 1 librados: init done 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d013cd80 0x7f55d002d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.222 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55d01248e0 con 0x7f55d013cd80 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d013cd80 0x7f55d002d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 10 monclient: get_auth_request con 0x7f55d013cd80 auth_method 0 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f55c00c4460 con 0x7f55d00a49a0 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 1 --2- 172.21.15.153:0/1298792648 >> v2:172.21.15.176:6812/3003473398 conn(0x7f55b003ce50 0x7f55b003f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 10 monclient(hunting): handle_auth_done global_id 34320 payload 824 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ced9d640 1 -- 172.21.15.153:0/1298792648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f55d00a3740 con 0x7f55d013cd80 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(301..301 src has 1..301) ==== 4803+0+0 (secure 0 0 0) 0x7f55c00eaaf0 con 0x7f55d00a49a0 2024-09-10T20:33:14.223 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 10 client.34317.objecter ms_dispatch 0x7f55d0000ba0 osd_map(301..301 src has 1..301) 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 3 client.34317.objecter handle_osd_map got epochs [301,301] > 0 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 3 client.34317.objecter handle_osd_map decoding full epoch 301 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55c77fe640 20 client.34317.objecter dump_active .. 0 homeless 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ce59c640 1 --2- 172.21.15.153:0/1298792648 >> v2:172.21.15.176:6812/3003473398 conn(0x7f55b003ce50 0x7f55b003f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.202+0000 7f55ce59c640 10 monclient: get_auth_request con 0x7f55b003ce50 auth_method 0 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 10 client.34317.objecter ms_handle_connect 0x7f55d013cd80 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 10 client.34317.objecter resend_mon_ops 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55ced9d640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d013cd80 msgr2=0x7f55d002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55ced9d640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d013cd80 0x7f55d002d450 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f55d0125960 tx=0x7f55c800db90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8003bd0 con 0x7f55d013cd80 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 10 monclient: dump: 2024-09-10T20:33:14.224 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f55c8017070 con 0x7f55d013cd80 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d013cd80 msgr2=0x7f55d002d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.225 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d013cd80 0x7f55d002d450 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f55d0125960 tx=0x7f55c800db90 comp rx=0 tx=0).stop 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d013cd80 msgr2=0x7f55d002d450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d013cd80 0x7f55d002d450 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f55d0125960 tx=0x7f55c800db90 comp rx=0 tx=0).stop 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55ce59c640 1 --2- 172.21.15.153:0/1298792648 >> v2:172.21.15.176:6812/3003473398 conn(0x7f55b003ce50 0x7f55b003f310 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7f55d00a0e70 tx=0x7f55b8007b30 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.206+0000 7f55c77fe640 10 client.34317.objecter ms_handle_connect 0x7f55b003ce50 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55d5037640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55d5037640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d0123880 0x7f55d002d990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55d5037640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f55d013f100 con 0x7f55d0123880 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d0123880 0x7f55d002d990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 10 monclient: get_auth_request con 0x7f55d0123880 auth_method 0 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:14.226 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 10 monclient(hunting): handle_auth_done global_id 34323 payload 824 2024-09-10T20:33:14.227 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.210+0000 7f55ced9d640 1 -- 172.21.15.153:0/1298792648 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f55d013d4e0 con 0x7f55d0123880 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55ced9d640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d0123880 msgr2=0x7f55d002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55ced9d640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d0123880 0x7f55d002d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f55c800b150 tx=0x7f55c80059e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 10 client.34317.objecter ms_handle_connect 0x7f55d0123880 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 10 client.34317.objecter resend_mon_ops 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c80052d0 con 0x7f55d0123880 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 10 monclient: dump: 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.228 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4711 (secure 0 0 0) 0x7f55c80052d0 con 0x7f55d0123880 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d0123880 msgr2=0x7f55d002d990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d0123880 0x7f55d002d990 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f55c800b150 tx=0x7f55c80059e0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.229 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d0123880 msgr2=0x7f55d002d990 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55c77fe640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d0123880 0x7f55d002d990 secure :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f55c800b150 tx=0x7f55c80059e0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 36, 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 154774, 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:33:14.230 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:33:14.231 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:33:14.232 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:33:14.233 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:33:14.234 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.235 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.236 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.237 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:33:14.238 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:33:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:33:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:33:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:33:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:33:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:33:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.239 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:33:14.240 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:33:14.241 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.242 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:33:14.243 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:33:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 10 librados: watch_flush enter 2024-09-10T20:33:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 10 librados: watch_flush exit 2024-09-10T20:33:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 20 client.34317.objecter shutdown clearing up homeless session... 2024-09-10T20:33:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 10 client.34317.objecter successfully canceled tick 2024-09-10T20:33:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 >> v2:172.21.15.176:6812/3003473398 conn(0x7f55b003ce50 msgr2=0x7f55b003f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 --2- 172.21.15.153:0/1298792648 >> v2:172.21.15.176:6812/3003473398 conn(0x7f55b003ce50 0x7f55b003f310 secure :-1 s=READY pgs=3360 cs=0 l=1 rev1=1 crypto rx=0x7f55d00a0e70 tx=0x7f55b8007b30 comp rx=0 tx=0).stop 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 10 monclient: shutdown 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d00a49a0 msgr2=0x7f55d01d05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d00a49a0 0x7f55d01d05c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f55c001c680 tx=0x7f55c00195f0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55cf59e640 1 -- 172.21.15.153:0/1298792648 reap_dead start 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 shutdown_connections 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 --2- 172.21.15.153:0/1298792648 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f55d00a49a0 0x7f55d01d05c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 >> 172.21.15.153:0/1298792648 conn(0x7f55d00ab7e0 msgr2=0x7f55d009ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 shutdown_connections 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 -- 172.21.15.153:0/1298792648 wait complete. 2024-09-10T20:33:14.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.214+0000 7f55d5037640 1 librados: shutdown 2024-09-10T20:33:14.246 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T20:33:14.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-10T20:33:14.246 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:33:14.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient: get_monmap_and_config 2024-09-10T20:33:14.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient: build_initial_monmap 2024-09-10T20:33:14.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient: monmap: 2024-09-10T20:33:14.359 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:33:14.358599+0000 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:33:14.358599+0000 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:33:14.360 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 1 Processor -- start 2024-09-10T20:33:14.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 1 -- start start 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient: init 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient: _add_conns ranks=[3,1,2,0,4] 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f825c1364f0 0x7f825c1368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient(hunting): picked mon.noname-a con 0x7f825c1364f0 addr v2:172.21.15.153:3300/0 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f825c1360a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c0a1990 0x7f825c09d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient(hunting): picked mon.noname-b con 0x7f825c0a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f825c0a27a0 con 0x7f825c1360a0 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f825c0a2a80 con 0x7f825c1364f0 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f825c0a2d60 con 0x7f825c0a1990 2024-09-10T20:33:14.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f8260c5d640 10 monclient(hunting): authenticate will time out at 1354.498265s 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825ad9d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c0a1990 msgr2=0x7f825c09d520 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c0a1990 0x7f825c09d520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f825c1364f0 0x7f825c1368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f825c1364f0 0x7f825c1368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39832/0 (socket says 172.21.15.153:39832) 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 1 -- 172.21.15.153:0/3584865289 learned_addr learned my addr 172.21.15.153:0/3584865289 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient(hunting): get_auth_request con 0x7f825c1364f0 auth_method 0 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:14.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:14.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient(hunting): handle_auth_done global_id 34326 payload 274 2024-09-10T20:33:14.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 1 -- 172.21.15.153:0/3584865289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c0a1990 msgr2=0x7f825c09d520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:33:14.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 1 --2- 172.21.15.153:0/3584865289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c0a1990 0x7f825c09d520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 1 -- 172.21.15.153:0/3584865289 >> v1:172.21.15.176:6789/0 conn(0x7f825c1360a0 legacy=0x7f825c09ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:33:14.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient: _finish_hunting 0 2024-09-10T20:33:14.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 1 monclient: found mon.noname-a 2024-09-10T20:33:14.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:14.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:33:14.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.354+0000 7f825a59c640 1 -- 172.21.15.153:0/3584865289 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c136e10 con 0x7f825c1364f0 2024-09-10T20:33:14.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825a59c640 1 -- 172.21.15.153:0/3584865289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1364f0 msgr2=0x7f825c1368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:14.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825a59c640 1 --2- 172.21.15.153:0/3584865289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1364f0 0x7f825c1368d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8254009870 tx=0x7f825401c440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f9ae5326611875b server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 1 -- 172.21.15.153:0/3584865289 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82540b1070 con 0x7f825c1364f0 2024-09-10T20:33:14.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:33:14.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 10 monclient: dump: 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 10 monclient: _finish_auth 0 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 10 monclient: _check_auth_tickets 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 1 -- 172.21.15.153:0/3584865289 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82540a8050 con 0x7f825c1364f0 2024-09-10T20:33:14.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 5 monclient: authenticate success, global_id 34326 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 10 monclient: get_monmap_and_config success 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 4 set_mon_vals no callback set 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 1 -- 172.21.15.153:0/3584865289 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82540ac860 con 0x7f825c1364f0 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f825959a640 10 monclient: dump: 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.371 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 10 monclient: shutdown 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 1 -- 172.21.15.153:0/3584865289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1364f0 msgr2=0x7f825c1368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 1 --2- 172.21.15.153:0/3584865289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1364f0 0x7f825c1368d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8254009870 tx=0x7f825401c440 comp rx=0 tx=0).stop 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 1 -- 172.21.15.153:0/3584865289 shutdown_connections 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 1 --2- 172.21.15.153:0/3584865289 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c0a1990 0x7f825c09d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 1 --2- 172.21.15.153:0/3584865289 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1364f0 0x7f825c1368d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 1 -- 172.21.15.153:0/3584865289 >> 172.21.15.153:0/3584865289 conn(0x7f825c0a71b0 msgr2=0x7f825c0a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 1 -- 172.21.15.153:0/3584865289 shutdown_connections 2024-09-10T20:33:14.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.358+0000 7f8260c5d640 1 -- 172.21.15.153:0/3584865289 wait complete. 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient: build_initial_monmap 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient: monmap: 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 librados: starting msgr at 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 librados: starting objecter 2024-09-10T20:33:14.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 Processor -- start 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 -- start start 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 librados: setting wanted keys 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 librados: calling monclient init 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient: init 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient: _reopen_session rank -1 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c0a1990 0x7f825c074240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient(hunting): picked mon.noname-b con 0x7f825c0a1990 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1360a0 0x7f825c09e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient(hunting): picked mon.noname-a con 0x7f825c1360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c1364f0 0x7f825c09e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient(hunting): picked mon.noname-c con 0x7f825c1364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f825c141040 con 0x7f825c1360a0 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f825c140ec0 con 0x7f825c0a1990 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f825c1411c0 con 0x7f825c1364f0 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient(hunting): _renew_subs 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8260c5d640 10 monclient(hunting): authenticate will time out at 1354.505452s 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f825ad9d640 1 -- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c1364f0 msgr2=0x7f825c09e820 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.153:3301/0 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f825ad9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c1364f0 0x7f825c09e820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1360a0 0x7f825c09e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1360a0 0x7f825c09e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:39834/0 (socket says 172.21.15.153:39834) 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 1 -- 172.21.15.153:0/3553802004 learned_addr learned my addr 172.21.15.153:0/3553802004 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient(hunting): get_auth_request con 0x7f825c1360a0 auth_method 0 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:14.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f825a59c640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c0a1990 0x7f825c074240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient(hunting): handle_auth_done global_id 34329 payload 824 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c1364f0 msgr2=0x7f825c09e820 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f825c1364f0 0x7f825c09e820 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c0a1990 msgr2=0x7f825c074240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c0a1990 0x7f825c074240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient: _finish_hunting 0 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 1 monclient: found mon.noname-a 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:33:14.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f8259d9b640 1 -- 172.21.15.153:0/3553802004 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f825c13df20 con 0x7f825c1360a0 2024-09-10T20:33:14.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f825a59c640 10 monclient: get_auth_request con 0x7f825c0a1990 auth_method 0 2024-09-10T20:33:14.377 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.362+0000 7f825a59c640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c0a1990 0x7f825c074240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:33:14.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8259d9b640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1360a0 msgr2=0x7f825c09e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:14.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8259d9b640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1360a0 0x7f825c09e2e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f824400c7c0 tx=0x7f824400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 client.?.objecter ms_handle_connect 0x7f825c1360a0 2024-09-10T20:33:14.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:33:14.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8259d9b640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1360a0 0x7f825c09e2e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f824400c7c0 tx=0x7f824400cc80 comp rx=0 tx=0).queue_received will delay after 1.726e+09 on 0x7f824400ac00 config(0 keys) 2024-09-10T20:33:14.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8244016020 con 0x7f825c1360a0 2024-09-10T20:33:14.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 monclient: dump: 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 monclient: _finish_auth 0 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:33:14.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f824400ac00 con 0x7f825c1360a0 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 5 monclient: authenticate success, global_id 34329 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 10 monclient: _renew_subs 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f825c13ceb0 con 0x7f825c1360a0 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 10 client.34329.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 10 monclient: _renew_subs 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f825c131690 con 0x7f825c1360a0 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 1 librados: init done 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8258d99640 4 set_mon_vals no callback set 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:14.382 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c02e5d0 0x7f825c09ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f825c02e9b0 con 0x7f825c02e5d0 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8244012420 con 0x7f825c1360a0 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 10 monclient: dump: 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.383 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f8244019020 con 0x7f825c1360a0 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c02e5d0 0x7f825c09ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f8252ffd640 1 --2- 172.21.15.153:0/3553802004 >> v2:172.21.15.176:6812/3003473398 conn(0x7f823c03ce50 0x7f823c03f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 10 monclient: get_auth_request con 0x7f825c02e5d0 auth_method 0 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825ad9d640 1 --2- 172.21.15.153:0/3553802004 >> v2:172.21.15.176:6812/3003473398 conn(0x7f823c03ce50 0x7f823c03f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825ad9d640 10 monclient: get_auth_request con 0x7f823c03ce50 auth_method 0 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 10 monclient(hunting): handle_auth_done global_id 20170 payload 824 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.366+0000 7f825a59c640 1 -- 172.21.15.153:0/3553802004 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f825c09e0a0 con 0x7f825c02e5d0 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f825a59c640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c02e5d0 msgr2=0x7f825c09ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f825a59c640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c02e5d0 0x7f825c09ed60 secure :-1 s=READY pgs=5741 cs=0 l=1 rev1=1 crypto rx=0x7f82540ae000 tx=0x7f825401ac80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 10 client.34329.objecter ms_handle_connect 0x7f825c02e5d0 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 10 client.34329.objecter resend_mon_ops 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 10 client.34329.objecter ms_handle_connect 0x7f823c03ce50 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f825ad9d640 1 --2- 172.21.15.153:0/3553802004 >> v2:172.21.15.176:6812/3003473398 conn(0x7f823c03ce50 0x7f823c03f310 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f824c003940 tx=0x7f824c006c50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82540bb020 con 0x7f825c02e5d0 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 10 monclient: dump: 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.385 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f82540a8040 con 0x7f825c02e5d0 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c02e5d0 msgr2=0x7f825c09ed60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c02e5d0 0x7f825c09ed60 secure :-1 s=READY pgs=5741 cs=0 l=1 rev1=1 crypto rx=0x7f82540ae000 tx=0x7f825401ac80 comp rx=0 tx=0).stop 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c02e5d0 msgr2=0x7f825c09ed60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:33:14.386 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.370+0000 7f8252ffd640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c02e5d0 0x7f825c09ed60 unknown :-1 s=CLOSED pgs=5741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.374+0000 7f8260c5d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:33:14.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.374+0000 7f8260c5d640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c063e00 0x7f825c0641e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:33:14.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.374+0000 7f8260c5d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:33:14.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.374+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f825c13d0f0 con 0x7f825c063e00 2024-09-10T20:33:14.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c063e00 0x7f825c0641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:33:14.388 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 10 monclient: get_auth_request con 0x7f825c063e00 auth_method 0 2024-09-10T20:33:14.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:33:14.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:33:14.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:33:14.389 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 10 monclient(hunting): handle_auth_done global_id 20173 payload 824 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 1 -- 172.21.15.153:0/3553802004 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f825c1368d0 con 0x7f825c063e00 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c063e00 msgr2=0x7f825c0641e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f825a59c640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c063e00 0x7f825c0641e0 secure :-1 s=READY pgs=5742 cs=0 l=1 rev1=1 crypto rx=0x7f825401c270 tx=0x7f82540191e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f8252ffd640 10 client.34329.objecter ms_handle_connect 0x7f825c063e00 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f8252ffd640 10 client.34329.objecter resend_mon_ops 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82540bb030 con 0x7f825c063e00 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f8252ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f8252ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.378+0000 7f8252ffd640 10 monclient: dump: 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:33:14.390 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4876 (secure 0 0 0) 0x7f825c09dc70 con 0x7f825c063e00 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c063e00 msgr2=0x7f825c0641e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c063e00 0x7f825c0641e0 secure :-1 s=READY pgs=5742 cs=0 l=1 rev1=1 crypto rx=0x7f825401c270 tx=0x7f82540191e0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c063e00 msgr2=0x7f825c0641e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:33:14.391 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f825c063e00 0x7f825c0641e0 secure :-1 s=CLOSED pgs=5742 cs=0 l=1 rev1=1 crypto rx=0x7f825401c270 tx=0x7f82540191e0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.392 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:33:14.392 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:33:14.392 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:33:14.392 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:33:14.392 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 36, 2024-09-10T20:33:14.392 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 441025, 2024-09-10T20:33:14.392 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:33:14.392 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:33:14.392 INFO:teuthology.orchestra.run.smithi153.stdout: 1 2024-09-10T20:33:14.393 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.393 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 0, 2024-09-10T20:33:14.393 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:33:14.393 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:33:14.393 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:33:14.393 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:33:14.393 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:33:14.393 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:33:14.394 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:33:14.395 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:33:14.396 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:33:14.397 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:33:14.398 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:33:14.399 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:33:14.400 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.401 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.402 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.403 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 6 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:33:14.404 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:33:14.405 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:33:14.405 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 10 librados: watch_flush enter 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 10 librados: watch_flush exit 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 20 client.34329.objecter shutdown clearing up homeless session... 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 10 client.34329.objecter successfully canceled tick 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 >> v2:172.21.15.176:6812/3003473398 conn(0x7f823c03ce50 msgr2=0x7f823c03f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 --2- 172.21.15.153:0/3553802004 >> v2:172.21.15.176:6812/3003473398 conn(0x7f823c03ce50 0x7f823c03f310 secure :-1 s=READY pgs=3361 cs=0 l=1 rev1=1 crypto rx=0x7f824c003940 tx=0x7f824c006c50 comp rx=0 tx=0).stop 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 10 monclient: shutdown 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1360a0 msgr2=0x7f825c09e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:33:14.407 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1360a0 0x7f825c09e2e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f824400c7c0 tx=0x7f824400cc80 comp rx=0 tx=0).stop 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f825ad9d640 1 -- 172.21.15.153:0/3553802004 reap_dead start 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 1 -- 172.21.15.153:0/3553802004 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(301..301 src has 1..301) ==== 4803+0+0 (unknown 0 0 0) 0x7f824404f9d0 con 0x7f825c1360a0 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 10 client.34329.objecter ms_dispatch 0x7f825c000ba0 osd_map(301..301 src has 1..301) 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8252ffd640 10 librados: disconnected, discarding osd_map(301..301 src has 1..301) 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 shutdown_connections 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 --2- 172.21.15.153:0/3553802004 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f825c1360a0 0x7f825c09e2e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 >> 172.21.15.153:0/3553802004 conn(0x7f825c0a71b0 msgr2=0x7f825c0a5170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 shutdown_connections 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 -- 172.21.15.153:0/3553802004 wait complete. 2024-09-10T20:33:14.408 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:33:14.382+0000 7f8260c5d640 1 librados: shutdown 2024-09-10T20:33:14.408 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-10T20:33:14.409 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-10T20:33:29.517 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-09-10T20:33:29.517 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (3026 ms) 2024-09-10T20:33:29.517 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-10T20:33:29.517 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (6958 ms) 2024-09-10T20:33:29.517 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-10T20:33:29.517 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7004 ms) 2024-09-10T20:33:29.518 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-10T20:33:29.518 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6973 ms) 2024-09-10T20:33:29.518 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-10T20:33:29.518 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (36339 ms) 2024-09-10T20:33:29.518 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-10T20:33:29.518 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6681 ms) 2024-09-10T20:33:29.518 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-10T20:33:29.518 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6995 ms) 2024-09-10T20:33:29.518 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (189071 ms total) 2024-09-10T20:33:29.519 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: 2024-09-10T20:33:29.519 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-10T20:33:29.519 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (373453 ms total) 2024-09-10T20:33:29.519 INFO:tasks.workunit.client.0.smithi176.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-10T20:33:29.525 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.525 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27952 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 27952 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28728 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28728 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28124 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28124 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28805 2024-09-10T20:33:29.526 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28805 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28212 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28212 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28591 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28591 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28860 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28860 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28162 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28162 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.527 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28432 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28432 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28696 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28696 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28495 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28495 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28783 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28783 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28407 2024-09-10T20:33:29.528 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28407 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28063 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28063 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28821 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28821 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28313 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28313 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27928 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 27928 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.529 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27968 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 27968 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28265 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28265 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=27988 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 27988 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28367 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28367 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28373 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28373 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.530 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28668 2024-09-10T20:33:29.531 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28668 2024-09-10T20:33:29.531 INFO:tasks.workunit.client.0.smithi176.stderr:+ for t in "${!pids[@]}" 2024-09-10T20:33:29.531 INFO:tasks.workunit.client.0.smithi176.stderr:+ pid=28882 2024-09-10T20:33:29.531 INFO:tasks.workunit.client.0.smithi176.stderr:+ wait 28882 2024-09-10T20:33:29.531 INFO:tasks.workunit.client.0.smithi176.stderr:+ exit 0 2024-09-10T20:33:29.531 INFO:tasks.workunit.client.0.smithi176.stderr:+ cleanup 2024-09-10T20:33:29.531 INFO:tasks.workunit.client.0.smithi176.stderr:+ pkill -P 27922 2024-09-10T20:33:29.532 INFO:tasks.workunit.client.0.smithi176.stderr:+ true 2024-09-10T20:33:29.533 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-10T20:33:29.533 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-10T20:33:29.547 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-09-10T20:33:29.548 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-10T20:33:29.908 DEBUG:teuthology.parallel:result is None 2024-09-10T20:33:29.909 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T20:33:29.921 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T20:33:29.922 DEBUG:teuthology.orchestra.run.smithi176:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-10T20:33:29.987 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-10T20:33:29.987 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-10T20:33:29.997 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-10T20:34:44.409 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-10T20:34:44.410 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-10T20:34:44.411 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-10T20:34:44.415 INFO:tasks.ceph.mon.c:Started 2024-09-10T20:34:44.415 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T20:34:44.415 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:34:44.510 INFO:tasks.ceph.mon.c.smithi153.stderr:2024-09-10T20:34:44.509+0000 7f870cc1be00 -1 mon.c@-1(???).log v0 log_external_backlog rewinding external_log_to from 293 -> 0 (sync_force? mon rebuild?) 2024-09-10T20:34:44.540 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient: get_monmap_and_config 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient: build_initial_monmap 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient: monmap: 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:44.540520+0000 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:44.540520+0000 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:44.541 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:44.542 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:44.542 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:44.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 1 Processor -- start 2024-09-10T20:34:44.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 1 -- start start 2024-09-10T20:34:44.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient: init 2024-09-10T20:34:44.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:44.544 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient: _add_conns ranks=[1,2,3,4,0] 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc8600a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 0x7fc86013d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient(hunting): picked mon.noname-b con 0x7fc8600a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc8600a49a0 0x7fc86013fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient(hunting): picked mon.noname-a con 0x7fc8600a49a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8600a6a20 con 0x7fc8600a41a0 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fc8600a2570 con 0x7fc8600a49a0 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc860070400 con 0x7fc8600a4580 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:44.545 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.537+0000 7fc8680b3640 10 monclient(hunting): authenticate will time out at 1444.679941s 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc8600a49a0 0x7fc86013fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 0x7fc86013d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 0x7fc86013d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:60290/0 (socket says 172.21.15.153:60290) 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fc8600a49a0 0x7fc86013fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47240/0 (socket says 172.21.15.153:47240) 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 1 -- 172.21.15.153:0/2600723188 learned_addr learned my addr 172.21.15.153:0/2600723188 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 10 monclient(hunting): get_auth_request con 0x7fc8600a4580 auth_method 0 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient(hunting): get_auth_request con 0x7fc8600a49a0 auth_method 0 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:44.546 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 1 -- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 msgr2=0x7fc86013d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 1 -- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 msgr2=0x7fc86013d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 1 --2- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 0x7fc86013d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc86564e640 1 --2- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 0x7fc86013d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient(hunting): handle_auth_done global_id 34338 payload 274 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 1 -- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 msgr2=0x7fc86013d7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 1 --2- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 0x7fc86013d7c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:44.547 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 1 -- 172.21.15.153:0/2600723188 >> v1:172.21.15.176:6789/0 conn(0x7fc8600a41a0 legacy=0x7fc8600a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:44.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient: _finish_hunting 0 2024-09-10T20:34:44.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 1 monclient: found mon.noname-a 2024-09-10T20:34:44.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:44.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:44.548 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.541+0000 7fc866650640 1 -- 172.21.15.153:0/2600723188 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8600a85a0 con 0x7fc8600a49a0 2024-09-10T20:34:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 1 -- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a49a0 msgr2=0x7fc86013fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:44.557 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 1 --2- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a49a0 0x7fc86013fd10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc8500097c0 tx=0x7fc85001c360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9ee66979d6be9ca server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 1 -- 172.21.15.153:0/2600723188 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8500b0070 con 0x7fc8600a49a0 2024-09-10T20:34:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 10 monclient: dump: 2024-09-10T20:34:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:44.558 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 10 monclient: _finish_auth 0 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 10 monclient: _check_auth_tickets 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 1 -- 172.21.15.153:0/2600723188 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc8500a9710 con 0x7fc8600a49a0 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 1 -- 172.21.15.153:0/2600723188 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8500a9ae0 con 0x7fc8600a49a0 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:44.559 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc864e4d640 10 monclient: dump: 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 5 monclient: authenticate success, global_id 34338 2024-09-10T20:34:44.560 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 4 set_mon_vals no callback set 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient: shutdown 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 -- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a49a0 msgr2=0x7fc86013fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 --2- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a49a0 0x7fc86013fd10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fc8500097c0 tx=0x7fc85001c360 comp rx=0 tx=0).stop 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 -- 172.21.15.153:0/2600723188 shutdown_connections 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 --2- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a49a0 0x7fc86013fd10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 --2- 172.21.15.153:0/2600723188 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a4580 0x7fc86013d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 -- 172.21.15.153:0/2600723188 >> 172.21.15.153:0/2600723188 conn(0x7fc8600ab7e0 msgr2=0x7fc8600a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 -- 172.21.15.153:0/2600723188 shutdown_connections 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 -- 172.21.15.153:0/2600723188 wait complete. 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient: build_initial_monmap 2024-09-10T20:34:44.561 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient: monmap: 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 librados: starting msgr at 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 librados: starting objecter 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 Processor -- start 2024-09-10T20:34:44.562 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 -- start start 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 librados: setting wanted keys 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 librados: calling monclient init 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient: init 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a41a0 0x7fc8601285a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient(hunting): picked mon.noname-a con 0x7fc8600a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8600a4580 0x7fc860126480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient(hunting): picked mon.noname-b con 0x7fc8600a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 0x7fc86013f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient(hunting): picked mon.noname-c con 0x7fc8600a49a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fc860142e00 con 0x7fc8600a41a0 2024-09-10T20:34:44.563 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fc860142c80 con 0x7fc8600a4580 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fc860142f80 con 0x7fc8600a49a0 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc8680b3640 10 monclient(hunting): authenticate will time out at 1444.687201s 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a41a0 0x7fc8601285a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 0x7fc86013f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a41a0 0x7fc8601285a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47246/0 (socket says 172.21.15.153:47246) 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 1 -- 172.21.15.153:0/304981351 learned_addr learned my addr 172.21.15.153:0/304981351 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 0x7fc86013f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:60296/0 (socket says 172.21.15.153:60296) 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient(hunting): get_auth_request con 0x7fc8600a41a0 auth_method 0 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:44.564 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 10 monclient(hunting): get_auth_request con 0x7fc8600a49a0 auth_method 0 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 1 -- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 msgr2=0x7fc86013f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 1 -- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 msgr2=0x7fc86013f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 0x7fc86013f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc866650640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 0x7fc86013f410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient(hunting): handle_auth_done global_id 34341 payload 824 2024-09-10T20:34:44.565 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 1 -- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 msgr2=0x7fc86013f410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 0x7fc86013f410 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 1 -- 172.21.15.153:0/304981351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8600a4580 msgr2=0x7fc860126480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc86564e640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8600a4580 0x7fc860126480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8600a4580 0x7fc860126480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient: _finish_hunting 0 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 1 monclient: found mon.noname-a 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.545+0000 7fc865e4f640 1 -- 172.21.15.153:0/304981351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc860127a40 con 0x7fc8600a41a0 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc865e4f640 1 -- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a41a0 msgr2=0x7fc8601285a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc865e4f640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a41a0 0x7fc8601285a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc85800c7c0 tx=0x7fc85800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 client.?.objecter ms_handle_connect 0x7fc8600a41a0 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:44.566 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 1 -- 172.21.15.153:0/304981351 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858016020 con 0x7fc8600a41a0 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 monclient: dump: 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:44.567 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 monclient: _finish_auth 0 2024-09-10T20:34:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:34:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 1 -- 172.21.15.153:0/304981351 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85800ac60 con 0x7fc8600a41a0 2024-09-10T20:34:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 1 -- 172.21.15.153:0/304981351 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858012660 con 0x7fc8600a41a0 2024-09-10T20:34:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 5 monclient: authenticate success, global_id 34341 2024-09-10T20:34:44.568 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 10 monclient: _renew_subs 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8600aa6a0 con 0x7fc8600a41a0 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 monclient: dump: 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:44.569 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85f7fe640 4 set_mon_vals no callback set 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 10 client.34341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 10 monclient: _renew_subs 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc860129da0 con 0x7fc8600a41a0 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 1 librados: init done 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:44.570 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc8601295c0 con 0x7fc8600a41a0 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 1 -- 172.21.15.153:0/304981351 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fc858019020 con 0x7fc8600a41a0 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 1 --2- 172.21.15.153:0/304981351 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc84c03cea0 0x7fc84c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 1 -- 172.21.15.153:0/304981351 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 1..317) ==== 4365+0+0 (secure 0 0 0) 0x7fc85804f680 con 0x7fc8600a41a0 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 10 client.34341.objecter ms_dispatch 0x7fc860000ba0 osd_map(317..317 src has 1..317) 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 3 client.34341.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 3 client.34341.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc85e7fc640 20 client.34341.objecter dump_active .. 0 homeless 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc86564e640 1 --2- 172.21.15.153:0/304981351 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc84c03cea0 0x7fc84c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.549+0000 7fc86564e640 10 monclient: get_auth_request con 0x7fc84c03cea0 auth_method 0 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.553+0000 7fc86564e640 1 --2- 172.21.15.153:0/304981351 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc84c03cea0 0x7fc84c03f360 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7fc8600aa430 tx=0x7fc854008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.553+0000 7fc85e7fc640 10 client.34341.objecter ms_handle_connect 0x7fc84c03cea0 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.557+0000 7fc85e7fc640 1 -- 172.21.15.153:0/304981351 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fc858020030 con 0x7fc8600a41a0 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.557+0000 7fc85e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:44.571 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.557+0000 7fc85e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:44.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.777+0000 7fc8680b3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:34:44.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.777+0000 7fc8680b3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:34:44.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.777+0000 7fc8680b3640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:44.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.777+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc8600a3720 con 0x7fc8600a41a0 2024-09-10T20:34:44.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.777+0000 7fc85e7fc640 1 -- 172.21.15.153:0/304981351 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1563 (secure 0 0 0) 0x7fc858012970 con 0x7fc8600a41a0 2024-09-10T20:34:44.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.777+0000 7fc85e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:34:44.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.777+0000 7fc85e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:34:44.781 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":36,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":91,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:34:44.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 10 librados: watch_flush enter 2024-09-10T20:34:44.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 10 librados: watch_flush exit 2024-09-10T20:34:44.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 20 client.34341.objecter shutdown clearing up homeless session... 2024-09-10T20:34:44.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 10 client.34341.objecter successfully canceled tick 2024-09-10T20:34:44.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc84c03cea0 msgr2=0x7fc84c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:44.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 --2- 172.21.15.153:0/304981351 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc84c03cea0 0x7fc84c03f360 secure :-1 s=READY pgs=3364 cs=0 l=1 rev1=1 crypto rx=0x7fc8600aa430 tx=0x7fc854008040 comp rx=0 tx=0).stop 2024-09-10T20:34:44.784 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 10 monclient: shutdown 2024-09-10T20:34:44.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:44.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a41a0 msgr2=0x7fc8601285a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:44.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a41a0 0x7fc8601285a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fc85800c7c0 tx=0x7fc85800cc80 comp rx=0 tx=0).stop 2024-09-10T20:34:44.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 shutdown_connections 2024-09-10T20:34:44.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 --2- 172.21.15.153:0/304981351 >> v2:172.21.15.176:6812/3003473398 conn(0x7fc84c03cea0 0x7fc84c03f360 unknown :-1 s=CLOSED pgs=3364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:44.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fc8600a49a0 0x7fc86013f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:44.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fc8600a4580 0x7fc860126480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:44.785 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 --2- 172.21.15.153:0/304981351 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fc8600a41a0 0x7fc8601285a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:44.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 >> 172.21.15.153:0/304981351 conn(0x7fc8600ab7e0 msgr2=0x7fc8600a0260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:44.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 shutdown_connections 2024-09-10T20:34:44.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 -- 172.21.15.153:0/304981351 wait complete. 2024-09-10T20:34:44.786 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:44.781+0000 7fc8680b3640 1 librados: shutdown 2024-09-10T20:34:47.799 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:34:47.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient: get_monmap_and_config 2024-09-10T20:34:47.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient: build_initial_monmap 2024-09-10T20:34:47.927 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient: monmap: 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:47.927439+0000 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:47.927439+0000 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:47.928 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:47.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 1 Processor -- start 2024-09-10T20:34:47.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 1 -- start start 2024-09-10T20:34:47.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient: init 2024-09-10T20:34:47.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:47.930 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient: _add_conns ranks=[2,0,1,3,4] 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 0x7f2da81368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient(hunting): picked mon.noname-b con 0x7f2da81364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2da81360a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2da80a1990 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da809dc30 con 0x7f2da81360a0 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2da80a27a0 con 0x7f2da80a1990 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2da80a2a80 con 0x7f2da81364f0 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:47.931 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2dae206640 10 monclient(hunting): authenticate will time out at 1448.067155s 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da77fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 0x7f2da81368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da6ffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f2da81360a0 0x7f2da809ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48322/0 (socket says 172.21.15.153:48322) 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da6ffd640 1 -- 172.21.15.153:0/2045048723 learned_addr learned my addr 172.21.15.153:0/2045048723 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da77fe640 10 monclient(hunting): get_auth_request con 0x7f2da81364f0 auth_method 0 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da77fe640 1 -- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 msgr2=0x7f2da81368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da77fe640 1 -- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 msgr2=0x7f2da81368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da77fe640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 0x7f2da81368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:47.932 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.925+0000 7f2da77fe640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 0x7f2da81368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:34:47.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 1 -- 172.21.15.153:0/2045048723 >> v1:172.21.15.153:6789/0 conn(0x7f2da81360a0 legacy=0x7f2da809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:47.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 <== mon.0 v1:172.21.15.153:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3019312142 0 0) 0x7f2da809dc30 con 0x7f2da81360a0 2024-09-10T20:34:47.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): _negotiate 2024-09-10T20:34:47.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): my global_id is 34344 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 --> v1:172.21.15.153:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d94003530 con 0x7f2da81360a0 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da7fff640 1 -- 172.21.15.153:0/2045048723 >> v1:172.21.15.176:6789/0 conn(0x7f2da80a1990 legacy=0x7f2da809d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 <== mon.0 v1:172.21.15.153:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1422812798 0 0) 0x7f2d94003530 con 0x7f2da81360a0 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 msgr2=0x7f2da81368d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 0x7f2da81368d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 >> v1:172.21.15.176:6789/0 conn(0x7f2da80a1990 legacy=0x7f2da809d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient: _finish_hunting 0 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 monclient: found mon.noname-a-legacy 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.153:6789/0 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 --> v1:172.21.15.153:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da80a2d60 con 0x7f2da81360a0 2024-09-10T20:34:47.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient: _finish_auth 0 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient: _check_auth_tickets 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 <== mon.0 v1:172.21.15.153:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 3765011595 0 0) 0x7f2d90001a70 con 0x7f2da81360a0 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2dae206640 5 monclient: authenticate success, global_id 34344 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2dae206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient: dump: 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:47.937 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 monclient: mon.a has (v2) addrs [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] but i'm connected to v1:172.21.15.153:6789/0, reconnecting 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 >> v1:172.21.15.153:6789/0 conn(0x7f2da81360a0 legacy=0x7f2da809ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d94003a40 0x7f2d94003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:47.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): picked mon.b con 0x7f2d94003a40 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 0x7f2d94004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): picked mon.c con 0x7f2da81364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d94004e70 0x7f2d9400b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): picked mon.a con 0x7f2d94004e70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2da80a2a80 con 0x7f2d94004e70 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2d94005630 con 0x7f2d94003a40 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2d9400bf10 con 0x7f2da81364f0 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da67fc640 10 monclient(hunting): _finish_auth 0 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2dae206640 20 monclient(hunting): get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:47.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 0x7f2d94004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient(hunting): get_auth_request con 0x7f2da81364f0 auth_method 0 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da7fff640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d94004e70 0x7f2d9400b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da7fff640 10 monclient(hunting): get_auth_request con 0x7f2d94004e70 auth_method 0 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da77fe640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d94003a40 0x7f2d94003e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:47.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da7fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da7fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da7fff640 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2024-09-10T20:34:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da77fe640 10 monclient(hunting): get_auth_request con 0x7f2d94003a40 auth_method 0 2024-09-10T20:34:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da77fe640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:34:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient(hunting): handle_auth_done global_id 34344 payload 293 2024-09-10T20:34:47.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 1 -- 172.21.15.153:0/2045048723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d94003a40 msgr2=0x7f2d94003e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2d94003a40 0x7f2d94003e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:47.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 1 -- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d94004e70 msgr2=0x7f2d9400b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2d94004e70 0x7f2d9400b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient: _finish_hunting 0 2024-09-10T20:34:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 1 monclient: found mon.c 2024-09-10T20:34:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 1 -- 172.21.15.153:0/2045048723 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d94003530 con 0x7f2da81364f0 2024-09-10T20:34:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient: _finish_auth 0 2024-09-10T20:34:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:34:47.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.929+0000 7f2da6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:47.949 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da6ffd640 1 -- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 msgr2=0x7f2d94004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da6ffd640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 0x7f2d94004810 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2d900014e0 tx=0x7f2d9001cd00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df81c5e183c94f15 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d9000c100 con 0x7f2da81364f0 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 10 monclient: dump: 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:47.950 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d900a9020 con 0x7f2da81364f0 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 1 -- 172.21.15.153:0/2045048723 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d900a5ca0 con 0x7f2da81364f0 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da67fc640 10 monclient: dump: 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.951 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:47.952 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 4 set_mon_vals no callback set 2024-09-10T20:34:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 10 monclient: shutdown 2024-09-10T20:34:47.952 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 1 -- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 msgr2=0x7f2d94004810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 1 --2- 172.21.15.153:0/2045048723 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da81364f0 0x7f2d94004810 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2d900014e0 tx=0x7f2d9001cd00 comp rx=0 tx=0).stop 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2da7fff640 1 -- 172.21.15.153:0/2045048723 reap_dead start 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 1 -- 172.21.15.153:0/2045048723 shutdown_connections 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 1 -- 172.21.15.153:0/2045048723 >> 172.21.15.153:0/2045048723 conn(0x7f2da80a71b0 msgr2=0x7f2da80a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 1 -- 172.21.15.153:0/2045048723 shutdown_connections 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.933+0000 7f2dae206640 1 -- 172.21.15.153:0/2045048723 wait complete. 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient: build_initial_monmap 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient: monmap: 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:47.953 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 librados: starting msgr at 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 librados: starting objecter 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 Processor -- start 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 -- start start 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 librados: setting wanted keys 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 librados: calling monclient init 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient: init 2024-09-10T20:34:47.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da81364f0 0x7f2da8073f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient(hunting): picked mon.noname-a con 0x7f2da81364f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da8074450 0x7f2da81ce610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient(hunting): picked mon.noname-c con 0x7f2da8074450 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2da81ceb50 0x7f2da81cefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient(hunting): picked mon.noname-b con 0x7f2da81ceb50 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f2da813fff0 con 0x7f2da81364f0 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f2da813fe70 con 0x7f2da81ceb50 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f2da8140170 con 0x7f2da8074450 2024-09-10T20:34:47.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2dae206640 10 monclient(hunting): authenticate will time out at 1448.077260s 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da81364f0 0x7f2da8073f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da81364f0 0x7f2da8073f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47300/0 (socket says 172.21.15.153:47300) 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 -- 172.21.15.153:0/3727485328 learned_addr learned my addr 172.21.15.153:0/3727485328 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient(hunting): get_auth_request con 0x7f2da81364f0 auth_method 0 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da6ffd640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da8074450 0x7f2da81ce610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:47.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da6ffd640 10 monclient(hunting): get_auth_request con 0x7f2da8074450 auth_method 0 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da6ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da6ffd640 1 -- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da8074450 msgr2=0x7f2da81ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da7fff640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2da81ceb50 0x7f2da81cefd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da6ffd640 1 -- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da8074450 msgr2=0x7f2da81ce610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da6ffd640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da8074450 0x7f2da81ce610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da7fff640 10 monclient(hunting): get_auth_request con 0x7f2da81ceb50 auth_method 0 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da7fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da7fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da6ffd640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da8074450 0x7f2da81ce610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient(hunting): handle_auth_done global_id 34347 payload 824 2024-09-10T20:34:47.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 -- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da8074450 msgr2=0x7f2da81ce610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da8074450 0x7f2da81ce610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 -- 172.21.15.153:0/3727485328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2da81ceb50 msgr2=0x7f2da81cefd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2da81ceb50 0x7f2da81cefd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient: _finish_hunting 0 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 monclient: found mon.noname-a 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 -- 172.21.15.153:0/3727485328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2da809e0a0 con 0x7f2da81364f0 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 -- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da81364f0 msgr2=0x7f2da8073f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2da77fe640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da81364f0 0x7f2da8073f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d980046b0 tx=0x7f2d98007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2d9ffff640 10 client.?.objecter ms_handle_connect 0x7f2da81364f0 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.937+0000 7f2d9ffff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:47.958 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 1 -- 172.21.15.153:0/3727485328 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98013780 con 0x7f2da81364f0 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 monclient: dump: 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 monclient: _finish_auth 0 2024-09-10T20:34:47.959 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 monclient: _check_auth_tickets 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 1 -- 172.21.15.153:0/3727485328 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d98019070 con 0x7f2da81364f0 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 5 monclient: authenticate success, global_id 34347 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 10 monclient: _renew_subs 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2da80717c0 con 0x7f2da81364f0 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 1 -- 172.21.15.153:0/3727485328 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d98013b40 con 0x7f2da81364f0 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 monclient: dump: 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:47.960 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2da5ffb640 4 set_mon_vals no callback set 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 10 client.34347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 10 monclient: _renew_subs 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2da80729e0 con 0x7f2da81364f0 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 1 librados: init done 2024-09-10T20:34:47.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2da809ce10 con 0x7f2da81364f0 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 1 -- 172.21.15.153:0/3727485328 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f2d98013ce0 con 0x7f2da81364f0 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 1 --2- 172.21.15.153:0/3727485328 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803ced0 0x7f2d8803f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 1 -- 172.21.15.153:0/3727485328 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 1..317) ==== 4365+0+0 (secure 0 0 0) 0x7f2d980500f0 con 0x7f2da81364f0 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 10 client.34347.objecter ms_dispatch 0x7f2da8000ba0 osd_map(317..317 src has 1..317) 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 3 client.34347.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 3 client.34347.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2d9ffff640 20 client.34347.objecter dump_active .. 0 homeless 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2da6ffd640 1 --2- 172.21.15.153:0/3727485328 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803ced0 0x7f2d8803f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.941+0000 7f2da6ffd640 10 monclient: get_auth_request con 0x7f2d8803ced0 auth_method 0 2024-09-10T20:34:47.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.945+0000 7f2d9ffff640 10 client.34347.objecter ms_handle_connect 0x7f2d8803ced0 2024-09-10T20:34:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.945+0000 7f2da6ffd640 1 --2- 172.21.15.153:0/3727485328 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803ced0 0x7f2d8803f390 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f2d90001ec0 tx=0x7f2da80a2c50 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.945+0000 7f2d9ffff640 1 -- 172.21.15.153:0/3727485328 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f2d98012ce0 con 0x7f2da81364f0 2024-09-10T20:34:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.945+0000 7f2d9ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:47.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:47.945+0000 7f2d9ffff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:34:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:34:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:48.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2da80711e0 con 0x7f2da81364f0 2024-09-10T20:34:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2d9ffff640 1 -- 172.21.15.153:0/3727485328 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f2d9801b310 con 0x7f2da81364f0 2024-09-10T20:34:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2d9ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:34:48.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2d9ffff640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:34:48.183 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:34:48.185 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 10 librados: watch_flush enter 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 10 librados: watch_flush exit 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 20 client.34347.objecter shutdown clearing up homeless session... 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 10 client.34347.objecter successfully canceled tick 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803ced0 msgr2=0x7f2d8803f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 --2- 172.21.15.153:0/3727485328 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803ced0 0x7f2d8803f390 secure :-1 s=READY pgs=3366 cs=0 l=1 rev1=1 crypto rx=0x7f2d90001ec0 tx=0x7f2da80a2c50 comp rx=0 tx=0).stop 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 10 monclient: shutdown 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da81364f0 msgr2=0x7f2da8073f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da81364f0 0x7f2da8073f10 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2d980046b0 tx=0x7f2d98007700 comp rx=0 tx=0).stop 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 shutdown_connections 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f2da81ceb50 0x7f2da81cefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.186 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 --2- 172.21.15.153:0/3727485328 >> v2:172.21.15.176:6812/3003473398 conn(0x7f2d8803ced0 0x7f2d8803f390 unknown :-1 s=CLOSED pgs=3366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f2da8074450 0x7f2da81ce610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 --2- 172.21.15.153:0/3727485328 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f2da81364f0 0x7f2da8073f10 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 >> 172.21.15.153:0/3727485328 conn(0x7f2da80a71b0 msgr2=0x7f2da8132170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:48.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 shutdown_connections 2024-09-10T20:34:48.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 -- 172.21.15.153:0/3727485328 wait complete. 2024-09-10T20:34:48.187 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.181+0000 7f2dae206640 1 librados: shutdown 2024-09-10T20:34:48.202 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T20:34:48.202 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-10T20:34:48.202 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-10T20:34:48.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient: get_monmap_and_config 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient: build_initial_monmap 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient: monmap: 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:48.322635+0000 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:48.322635+0000 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:48.323 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:48.324 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:48.324 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 1 Processor -- start 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 1 -- start start 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient: init 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient: _add_conns ranks=[4,2,3,0,1] 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f71cc0a41a0 0x7f71cc0a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient(hunting): picked mon.noname-c con 0x7f71cc0a41a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a4580 0x7f71cc0a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient(hunting): picked mon.noname-b con 0x7f71cc0a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f71cc0a49a0 0x7f71cc0a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.326 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient(hunting): picked mon.noname-a con 0x7f71cc0a49a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f71cc0a2570 con 0x7f71cc0a49a0 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f71cc070400 con 0x7f71cc0a41a0 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f71cc0a6a20 con 0x7f71cc0a4580 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d3ab0640 10 monclient(hunting): authenticate will time out at 1448.462304s 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f71cc0a49a0 0x7f71cc0a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d104b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a4580 0x7f71cc0a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f71cc0a49a0 0x7f71cc0a5f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47306/0 (socket says 172.21.15.153:47306) 2024-09-10T20:34:48.327 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 1 -- 172.21.15.153:0/2184129356 learned_addr learned my addr 172.21.15.153:0/2184129356 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d104b640 10 monclient(hunting): get_auth_request con 0x7f71cc0a4580 auth_method 0 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d104b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d104b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d104b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient(hunting): get_auth_request con 0x7f71cc0a49a0 auth_method 0 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d104b640 1 -- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a4580 msgr2=0x7f71cc0a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d104b640 1 -- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a4580 msgr2=0x7f71cc0a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-09-10T20:34:48.328 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d104b640 1 --2- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a4580 0x7f71cc0a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d104b640 1 --2- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a4580 0x7f71cc0a59d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d184c640 1 --2- 172.21.15.153:0/2184129356 >> v2:172.21.15.176:3300/0 conn(0x7f71cc0a41a0 0x7f71cc0a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient(hunting): handle_auth_done global_id 34350 payload 274 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 1 -- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a4580 msgr2=0x7f71cc0a59d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 1 --2- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a4580 0x7f71cc0a59d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 1 -- 172.21.15.153:0/2184129356 >> v2:172.21.15.176:3300/0 conn(0x7f71cc0a41a0 msgr2=0x7f71cc0a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 1 --2- 172.21.15.153:0/2184129356 >> v2:172.21.15.176:3300/0 conn(0x7f71cc0a41a0 0x7f71cc0a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient: _finish_hunting 0 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 1 monclient: found mon.noname-a 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:48.329 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d204d640 1 -- 172.21.15.153:0/2184129356 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc0a3c80 con 0x7f71cc0a49a0 2024-09-10T20:34:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d184c640 10 monclient: get_auth_request con 0x7f71cc0a41a0 auth_method 0 2024-09-10T20:34:48.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.321+0000 7f71d184c640 1 --2- 172.21.15.153:0/2184129356 >> v2:172.21.15.176:3300/0 conn(0x7f71cc0a41a0 0x7f71cc0a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:48.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d204d640 1 -- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a49a0 msgr2=0x7f71cc0a5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:48.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d204d640 1 --2- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a49a0 0x7f71cc0a5f10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0a8a30 tx=0x7f71c401c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0a163622485a94a server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:48.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 1 -- 172.21.15.153:0/2184129356 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40ac020 con 0x7f71cc0a49a0 2024-09-10T20:34:48.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:48.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 10 monclient: dump: 2024-09-10T20:34:48.336 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.336 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.336 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 10 monclient: _finish_auth 0 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 10 monclient: _check_auth_tickets 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 1 -- 172.21.15.153:0/2184129356 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71c40a6430 con 0x7f71cc0a49a0 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:48.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 1 -- 172.21.15.153:0/2184129356 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c40a6800 con 0x7f71cc0a49a0 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 5 monclient: authenticate success, global_id 34350 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 4 set_mon_vals no callback set 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d084a640 10 monclient: dump: 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.338 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 10 monclient: shutdown 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 1 -- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a49a0 msgr2=0x7f71cc0a5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 1 --2- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a49a0 0x7f71cc0a5f10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0a8a30 tx=0x7f71c401c8b0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 1 -- 172.21.15.153:0/2184129356 shutdown_connections 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 1 --2- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a49a0 0x7f71cc0a5f10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 1 --2- 172.21.15.153:0/2184129356 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a4580 0x7f71cc0a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 1 --2- 172.21.15.153:0/2184129356 >> v2:172.21.15.176:3300/0 conn(0x7f71cc0a41a0 0x7f71cc0a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 1 -- 172.21.15.153:0/2184129356 >> 172.21.15.153:0/2184129356 conn(0x7f71cc0ab7e0 msgr2=0x7f71cc0a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 1 -- 172.21.15.153:0/2184129356 shutdown_connections 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.325+0000 7f71d3ab0640 1 -- 172.21.15.153:0/2184129356 wait complete. 2024-09-10T20:34:48.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient: build_initial_monmap 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient: monmap: 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 librados: starting msgr at 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 librados: starting objecter 2024-09-10T20:34:48.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 Processor -- start 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 -- start start 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 librados: setting wanted keys 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 librados: calling monclient init 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient: init 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc0a41a0 0x7f71cc1d04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient(hunting): picked mon.noname-b con 0x7f71cc0a41a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a4580 0x7f71cc13e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient(hunting): picked mon.noname-a con 0x7f71cc0a4580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a49a0 0x7f71cc126400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient(hunting): picked mon.noname-c con 0x7f71cc0a49a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:48.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc142200 con 0x7f71cc0a4580 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc0a6860 con 0x7f71cc0a41a0 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f71cc142380 con 0x7f71cc0a49a0 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d3ab0640 10 monclient(hunting): authenticate will time out at 1448.469368s 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a4580 0x7f71cc13e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a4580 0x7f71cc13e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47312/0 (socket says 172.21.15.153:47312) 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 1 -- 172.21.15.153:0/525703536 learned_addr learned my addr 172.21.15.153:0/525703536 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d204d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a49a0 0x7f71cc126400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient(hunting): get_auth_request con 0x7f71cc0a4580 auth_method 0 2024-09-10T20:34:48.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d204d640 10 monclient(hunting): get_auth_request con 0x7f71cc0a49a0 auth_method 0 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d204d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d204d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d204d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d204d640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a49a0 msgr2=0x7f71cc126400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d204d640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a49a0 msgr2=0x7f71cc126400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d204d640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a49a0 0x7f71cc126400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:48.343 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d204d640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a49a0 0x7f71cc126400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d184c640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc0a41a0 0x7f71cc1d04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d184c640 10 monclient(hunting): get_auth_request con 0x7f71cc0a41a0 auth_method 0 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d184c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d184c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d184c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient(hunting): handle_auth_done global_id 34353 payload 824 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a49a0 msgr2=0x7f71cc126400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f71cc0a49a0 0x7f71cc126400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc0a41a0 msgr2=0x7f71cc1d04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f71cc0a41a0 0x7f71cc1d04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient: _finish_hunting 0 2024-09-10T20:34:48.344 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 1 monclient: found mon.noname-a 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.329+0000 7f71d104b640 1 -- 172.21.15.153:0/525703536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71cc13f580 con 0x7f71cc0a4580 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d104b640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a4580 msgr2=0x7f71cc13e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d104b640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a4580 0x7f71cc13e7e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00e7e0 tx=0x7f71bc00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 client.?.objecter ms_handle_connect 0x7f71cc0a4580 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc017020 con 0x7f71cc0a4580 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 monclient: dump: 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 monclient: _finish_auth 0 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 monclient: _check_auth_tickets 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71bc00c8f0 con 0x7f71cc0a4580 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71bc00f040 con 0x7f71cc0a4580 2024-09-10T20:34:48.346 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 5 monclient: authenticate success, global_id 34353 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 10 monclient: _renew_subs 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71cc1d13c0 con 0x7f71cc0a4580 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c37fe640 4 set_mon_vals no callback set 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 monclient: dump: 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.347 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 10 client.34353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 10 monclient: _renew_subs 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71cc129bc0 con 0x7f71cc0a4580 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 1 librados: init done 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f71bc01a020 con 0x7f71cc0a4580 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc128980 0x7f71cc126940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc129e00 con 0x7f71cc128980 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 1 --2- 172.21.15.153:0/525703536 >> v2:172.21.15.176:6812/3003473398 conn(0x7f71b003cea0 0x7f71b003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc128980 0x7f71cc126940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 10 monclient: get_auth_request con 0x7f71cc128980 auth_method 0 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 1..317) ==== 4365+0+0 (secure 0 0 0) 0x7f71bc04f4c0 con 0x7f71cc0a4580 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 10 client.34353.objecter ms_dispatch 0x7f71cc000ba0 osd_map(317..317 src has 1..317) 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 3 client.34353.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 3 client.34353.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:48.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:48.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71c27fc640 20 client.34353.objecter dump_active .. 0 homeless 2024-09-10T20:34:48.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:48.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 10 monclient(hunting): handle_auth_done global_id 34356 payload 824 2024-09-10T20:34:48.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d204d640 1 --2- 172.21.15.153:0/525703536 >> v2:172.21.15.176:6812/3003473398 conn(0x7f71b003cea0 0x7f71b003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d184c640 1 -- 172.21.15.153:0/525703536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71cc0a3720 con 0x7f71cc128980 2024-09-10T20:34:48.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.333+0000 7f71d204d640 10 monclient: get_auth_request con 0x7f71b003cea0 auth_method 0 2024-09-10T20:34:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 10 client.34353.objecter ms_handle_connect 0x7f71cc128980 2024-09-10T20:34:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 10 client.34353.objecter resend_mon_ops 2024-09-10T20:34:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71d184c640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc128980 msgr2=0x7f71cc126940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71d184c640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc128980 0x7f71cc126940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f71c800f4b0 tx=0x7f71c800f970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8018070 con 0x7f71cc128980 2024-09-10T20:34:48.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 10 monclient: dump: 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71c8004220 con 0x7f71cc128980 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc128980 msgr2=0x7f71cc126940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc128980 0x7f71cc126940 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f71c800f4b0 tx=0x7f71c800f970 comp rx=0 tx=0).stop 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc128980 msgr2=0x7f71cc126940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc128980 0x7f71cc126940 secure :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f71c800f4b0 tx=0x7f71c800f970 comp rx=0 tx=0).stop 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71c27fc640 10 client.34353.objecter ms_handle_connect 0x7f71b003cea0 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.337+0000 7f71d204d640 1 --2- 172.21.15.153:0/525703536 >> v2:172.21.15.176:6812/3003473398 conn(0x7f71b003cea0 0x7f71b003f360 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0a7aa0 tx=0x7f71c40a7000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d3ab0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d3ab0640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc13ed20 0x7f71cc126e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d3ab0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.354 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f71cc06cd40 con 0x7f71cc13ed20 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc13ed20 0x7f71cc126e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 10 monclient: get_auth_request con 0x7f71cc13ed20 auth_method 0 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 10 monclient(hunting): handle_auth_done global_id 34359 payload 824 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.341+0000 7f71d184c640 1 -- 172.21.15.153:0/525703536 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71cc1db820 con 0x7f71cc13ed20 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 10 client.34353.objecter ms_handle_connect 0x7f71cc13ed20 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d184c640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc13ed20 msgr2=0x7f71cc126e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:48.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 10 client.34353.objecter resend_mon_ops 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d184c640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc13ed20 0x7f71cc126e80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f71c8019000 tx=0x7f71c80046d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71c8018040 con 0x7f71cc13ed20 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 10 monclient: dump: 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.356 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.357 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 <== mon.0 v2:172.21.15.153:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4722 (secure 0 0 0) 0x7f71c8003da0 con 0x7f71cc13ed20 2024-09-10T20:34:48.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:34:48.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:34:48.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc13ed20 msgr2=0x7f71cc126e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc13ed20 0x7f71cc126e80 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f71c8019000 tx=0x7f71c80046d0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc13ed20 msgr2=0x7f71cc126e80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:34:48.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71c27fc640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc13ed20 0x7f71cc126e80 secure :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f71c8019000 tx=0x7f71c80046d0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.357 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "leader", 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 38, 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 248905, 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:48.358 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 1, 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:34:48.359 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:34:48.360 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:34:48.361 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:34:48.362 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:34:48.363 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:34:48.364 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:34:48.365 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:34:48.366 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:48.367 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:48.368 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:34:48.369 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 2 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:34:48.370 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:34:48.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 10 librados: watch_flush enter 2024-09-10T20:34:48.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 10 librados: watch_flush exit 2024-09-10T20:34:48.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 20 client.34353.objecter shutdown clearing up homeless session... 2024-09-10T20:34:48.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 10 client.34353.objecter successfully canceled tick 2024-09-10T20:34:48.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 >> v2:172.21.15.176:6812/3003473398 conn(0x7f71b003cea0 msgr2=0x7f71b003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 --2- 172.21.15.153:0/525703536 >> v2:172.21.15.176:6812/3003473398 conn(0x7f71b003cea0 0x7f71b003f360 secure :-1 s=READY pgs=3367 cs=0 l=1 rev1=1 crypto rx=0x7f71cc0a7aa0 tx=0x7f71c40a7000 comp rx=0 tx=0).stop 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 10 monclient: shutdown 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a4580 msgr2=0x7f71cc13e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a4580 0x7f71cc13e7e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f71bc00e7e0 tx=0x7f71bc00eca0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d204d640 1 -- 172.21.15.153:0/525703536 reap_dead start 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 shutdown_connections 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 --2- 172.21.15.153:0/525703536 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f71cc0a4580 0x7f71cc13e7e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 >> 172.21.15.153:0/525703536 conn(0x7f71cc0ab7e0 msgr2=0x7f71cc09ca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 shutdown_connections 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 -- 172.21.15.153:0/525703536 wait complete. 2024-09-10T20:34:48.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.345+0000 7f71d3ab0640 1 librados: shutdown 2024-09-10T20:34:48.374 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:34:48.374 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-10T20:34:48.374 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-10T20:34:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient: get_monmap_and_config 2024-09-10T20:34:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient: build_initial_monmap 2024-09-10T20:34:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient: monmap: 2024-09-10T20:34:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:48.486667+0000 2024-09-10T20:34:48.490 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:48.486667+0000 2024-09-10T20:34:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:48.491 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 1 Processor -- start 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 1 -- start start 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient: init 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient: _add_conns ranks=[4,1,3,0,2] 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f40341364f0 0x7f40341368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient(hunting): picked mon.noname-c con 0x7f40341364f0 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f40341360a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f40340a1990 0x7f403409d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient(hunting): picked mon.noname-a con 0x7f40340a1990 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40340a27a0 con 0x7f40341360a0 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f40340a2a80 con 0x7f40340a1990 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f40340a2d60 con 0x7f40341364f0 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f40390ff640 10 monclient(hunting): authenticate will time out at 1448.626402s 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f40340a1990 0x7f403409d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f40340a1990 0x7f403409d520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47338/0 (socket says 172.21.15.153:47338) 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 1 -- 172.21.15.153:0/1212657071 learned_addr learned my addr 172.21.15.153:0/1212657071 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:48.493 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient(hunting): get_auth_request con 0x7f40340a1990 auth_method 0 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f4032d9d640 1 --2- 172.21.15.153:0/1212657071 >> v2:172.21.15.176:3300/0 conn(0x7f40341364f0 0x7f40341368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f4032d9d640 10 monclient(hunting): get_auth_request con 0x7f40341364f0 auth_method 0 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f4032d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f4032d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.494 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f4032d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient(hunting): handle_auth_done global_id 34362 payload 274 2024-09-10T20:34:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 1 -- 172.21.15.153:0/1212657071 >> v2:172.21.15.176:3300/0 conn(0x7f40341364f0 msgr2=0x7f40341368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 1 --2- 172.21.15.153:0/1212657071 >> v2:172.21.15.176:3300/0 conn(0x7f40341364f0 0x7f40341368d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 1 -- 172.21.15.153:0/1212657071 >> v1:172.21.15.176:6789/0 conn(0x7f40341360a0 legacy=0x7f403409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient: _finish_hunting 0 2024-09-10T20:34:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 1 monclient: found mon.noname-a 2024-09-10T20:34:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:48.495 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.485+0000 7f403359e640 1 -- 172.21.15.153:0/1212657071 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4034136e10 con 0x7f40340a1990 2024-09-10T20:34:48.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f403359e640 1 -- 172.21.15.153:0/1212657071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 msgr2=0x7f403409d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:48.502 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f403359e640 1 --2- 172.21.15.153:0/1212657071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 0x7f403409d520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f40340a12f0 tx=0x7f402c01c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399263ea8d729fa3 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 1 -- 172.21.15.153:0/1212657071 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c0a5750 con 0x7f40340a1990 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 10 monclient: dump: 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.503 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 10 monclient: _finish_auth 0 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 10 monclient: _check_auth_tickets 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 1 -- 172.21.15.153:0/1212657071 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f403409dc70 con 0x7f40340a1990 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 1 -- 172.21.15.153:0/1212657071 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402c0a5b10 con 0x7f40340a1990 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:48.504 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f4031d9b640 10 monclient: dump: 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f40390ff640 5 monclient: authenticate success, global_id 34362 2024-09-10T20:34:48.505 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f40390ff640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:48.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.489+0000 7f40390ff640 4 set_mon_vals no callback set 2024-09-10T20:34:48.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient: shutdown 2024-09-10T20:34:48.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:48.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 -- 172.21.15.153:0/1212657071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 msgr2=0x7f403409d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 --2- 172.21.15.153:0/1212657071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 0x7f403409d520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f40340a12f0 tx=0x7f402c01c8b0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 -- 172.21.15.153:0/1212657071 shutdown_connections 2024-09-10T20:34:48.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 --2- 172.21.15.153:0/1212657071 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 0x7f403409d520 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 --2- 172.21.15.153:0/1212657071 >> v2:172.21.15.176:3300/0 conn(0x7f40341364f0 0x7f40341368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.506 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 -- 172.21.15.153:0/1212657071 >> 172.21.15.153:0/1212657071 conn(0x7f40340a71b0 msgr2=0x7f40340a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 -- 172.21.15.153:0/1212657071 shutdown_connections 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 -- 172.21.15.153:0/1212657071 wait complete. 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient: build_initial_monmap 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient: monmap: 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.507 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 librados: starting msgr at 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 librados: starting objecter 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 Processor -- start 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 -- start start 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 librados: setting wanted keys 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 librados: calling monclient init 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient: init 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:48.508 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 0x7f403413d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient(hunting): picked mon.noname-a con 0x7f40340a1990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341360a0 0x7f40341378e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient(hunting): picked mon.noname-c con 0x7f40341360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40341364f0 0x7f4034137e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient(hunting): picked mon.noname-b con 0x7f40341364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f4034140ff0 con 0x7f40340a1990 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f4034140e70 con 0x7f40341364f0 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.509 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f4034141170 con 0x7f40341360a0 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f40390ff640 10 monclient(hunting): authenticate will time out at 1448.634016s 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 0x7f403413d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 0x7f403413d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47354/0 (socket says 172.21.15.153:47354) 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 1 -- 172.21.15.153:0/3341061564 learned_addr learned my addr 172.21.15.153:0/3341061564 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient(hunting): get_auth_request con 0x7f40340a1990 auth_method 0 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341360a0 0x7f40341378e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.510 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403259c640 10 monclient(hunting): get_auth_request con 0x7f40341360a0 auth_method 0 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341360a0 msgr2=0x7f40341378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341360a0 msgr2=0x7f40341378e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341360a0 0x7f40341378e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341360a0 0x7f40341378e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403359e640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40341364f0 0x7f4034137e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.511 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient(hunting): handle_auth_done global_id 34365 payload 824 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341360a0 msgr2=0x7f40341378e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341360a0 0x7f40341378e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40341364f0 msgr2=0x7f4034137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40341364f0 0x7f4034137e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient: _finish_hunting 0 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 1 monclient: found mon.noname-a 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f4032d9d640 1 -- 172.21.15.153:0/3341061564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4034130d00 con 0x7f40340a1990 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403359e640 10 monclient: get_auth_request con 0x7f40341364f0 auth_method 0 2024-09-10T20:34:48.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.493+0000 7f403359e640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f40341364f0 0x7f4034137e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f4032d9d640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 msgr2=0x7f403413d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f4032d9d640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 0x7f403413d8b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f402800d800 tx=0x7f402800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 client.?.objecter ms_handle_connect 0x7f40340a1990 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402800b8d0 con 0x7f40340a1990 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 monclient: dump: 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.513 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4028019070 con 0x7f40340a1990 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:48.514 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f402800bc70 con 0x7f40340a1990 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 5 monclient: authenticate success, global_id 34365 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 10 monclient: _renew_subs 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40340742c0 con 0x7f40340a1990 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403159a640 4 set_mon_vals no callback set 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 monclient: dump: 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.515 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 10 client.34365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 10 monclient: _renew_subs 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f403413b290 con 0x7f40340a1990 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 1 librados: init done 2024-09-10T20:34:48.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f403409dc70 con 0x7f4034138360 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403259c640 10 monclient: get_auth_request con 0x7f4034138360 auth_method 0 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f402801e460 con 0x7f40340a1990 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 msgr2=0x7f4034133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 msgr2=0x7f4034133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:34:48.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:48.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-09-10T20:34:48.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 1 --2- 172.21.15.153:0/3341061564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f402003ced0 0x7f402003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:48.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 1..317) ==== 4365+0+0 (secure 0 0 0) 0x7f402804fb10 con 0x7f40340a1990 2024-09-10T20:34:48.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 10 client.34365.objecter ms_dispatch 0x7f4034000ba0 osd_map(317..317 src has 1..317) 2024-09-10T20:34:48.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 3 client.34365.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:48.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 3 client.34365.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:48.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f401b7fe640 20 client.34365.objecter dump_active .. 0 homeless 2024-09-10T20:34:48.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403359e640 1 --2- 172.21.15.153:0/3341061564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f402003ced0 0x7f402003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.497+0000 7f403359e640 10 monclient: get_auth_request con 0x7f402003ced0 auth_method 0 2024-09-10T20:34:48.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.501+0000 7f403359e640 1 --2- 172.21.15.153:0/3341061564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f402003ced0 0x7f402003f390 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f40340a13d0 tx=0x7f402c0a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:48.519 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.501+0000 7f401b7fe640 10 client.34365.objecter ms_handle_connect 0x7f402003ced0 2024-09-10T20:34:48.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.701+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:48.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.701+0000 7f403259c640 10 monclient: get_auth_request con 0x7f4034138360 auth_method 0 2024-09-10T20:34:48.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.701+0000 7f403259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:48.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.701+0000 7f403259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:48.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.701+0000 7f403259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:34:48.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.701+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 msgr2=0x7f4034133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T20:34:48.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.701+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 msgr2=0x7f4034133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:34:48.703 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.701+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:48.704 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:48.701+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-09-10T20:34:49.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.105+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:49.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.105+0000 7f403259c640 10 monclient: get_auth_request con 0x7f4034138360 auth_method 0 2024-09-10T20:34:49.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.105+0000 7f403259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:49.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.105+0000 7f403259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:49.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.105+0000 7f403259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:34:49.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.105+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 msgr2=0x7f4034133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-10T20:34:49.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.105+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 msgr2=0x7f4034133c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-10T20:34:49.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.105+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-09-10T20:34:49.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.105+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 10 monclient: get_auth_request con 0x7f4034138360 auth_method 0 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 10 monclient(hunting): handle_auth_done global_id 54101 payload 824 2024-09-10T20:34:49.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.909+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4034138740 con 0x7f4034138360 2024-09-10T20:34:49.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 msgr2=0x7f4034133c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:49.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 10 client.34365.objecter ms_handle_connect 0x7f4034138360 2024-09-10T20:34:49.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 10 client.34365.objecter resend_mon_ops 2024-09-10T20:34:49.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4024014510 tx=0x7f40240149d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:49.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4024005490 con 0x7f4034138360 2024-09-10T20:34:49.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:49.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:49.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 10 monclient: dump: 2024-09-10T20:34:49.915 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:49.915 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:49.915 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:49.916 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:49.916 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:49.916 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:49.916 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:49.916 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:49.916 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:49.916 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:49.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4024005490 con 0x7f4034138360 2024-09-10T20:34:49.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:49.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:49.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 msgr2=0x7f4034133c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:49.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4024014510 tx=0x7f40240149d0 comp rx=0 tx=0).stop 2024-09-10T20:34:49.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 msgr2=0x7f4034133c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:34:49.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.913+0000 7f401b7fe640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f4034138360 0x7f4034133c70 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4024014510 tx=0x7f40240149d0 comp rx=0 tx=0).stop 2024-09-10T20:34:49.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f40390ff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:34:49.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f40390ff640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341341b0 0x7f40341345c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:49.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f40390ff640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:49.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f403413ae40 con 0x7f40341341b0 2024-09-10T20:34:49.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341341b0 0x7f40341345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:49.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 10 monclient: get_auth_request con 0x7f40341341b0 auth_method 0 2024-09-10T20:34:49.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:49.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:49.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:49.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:49.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:49.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:49.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 10 monclient(hunting): handle_auth_done global_id 54104 payload 824 2024-09-10T20:34:49.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.921+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f40341368d0 con 0x7f40341341b0 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f403259c640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341341b0 msgr2=0x7f40341345c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 10 client.34365.objecter ms_handle_connect 0x7f40341341b0 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f403259c640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341341b0 0x7f40341345c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4024014510 tx=0x7f4024003700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 10 client.34365.objecter resend_mon_ops 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4024006bf0 con 0x7f40341341b0 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 10 monclient: dump: 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:49.926 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:49.927 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:49.927 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:49.927 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:49.927 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:49.927 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:49.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 <== mon.2 v2:172.21.15.153:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f4024007670 con 0x7f40341341b0 2024-09-10T20:34:49.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:34:49.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:34:49.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341341b0 msgr2=0x7f40341345c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:49.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341341b0 0x7f40341345c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4024014510 tx=0x7f4024003700 comp rx=0 tx=0).stop 2024-09-10T20:34:49.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341341b0 msgr2=0x7f40341345c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:34:49.928 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f401b7fe640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f40341341b0 0x7f40341345c0 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4024014510 tx=0x7f4024003700 comp rx=0 tx=0).stop 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 38, 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 5431, 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:34:49.929 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:49.930 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:34:49.931 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:34:49.932 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:34:49.933 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:34:49.934 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:34:49.935 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:34:49.936 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:34:49.937 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:49.938 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:34:49.939 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:34:49.940 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:34:49.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 10 librados: watch_flush enter 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 10 librados: watch_flush exit 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 20 client.34365.objecter shutdown clearing up homeless session... 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 10 client.34365.objecter successfully canceled tick 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f402003ced0 msgr2=0x7f402003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 --2- 172.21.15.153:0/3341061564 >> v2:172.21.15.176:6812/3003473398 conn(0x7f402003ced0 0x7f402003f390 secure :-1 s=READY pgs=3368 cs=0 l=1 rev1=1 crypto rx=0x7f40340a13d0 tx=0x7f402c0a7070 comp rx=0 tx=0).stop 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 10 monclient: shutdown 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 msgr2=0x7f403413d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 0x7f403413d8b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f402800d800 tx=0x7f402800dcc0 comp rx=0 tx=0).stop 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f403359e640 1 -- 172.21.15.153:0/3341061564 reap_dead start 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 shutdown_connections 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 --2- 172.21.15.153:0/3341061564 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f40340a1990 0x7f403413d8b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 >> 172.21.15.153:0/3341061564 conn(0x7f40340a71b0 msgr2=0x7f403412ed10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:49.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 shutdown_connections 2024-09-10T20:34:49.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 -- 172.21.15.153:0/3341061564 wait complete. 2024-09-10T20:34:49.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:49.925+0000 7f40390ff640 1 librados: shutdown 2024-09-10T20:34:49.947 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:34:49.947 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-10T20:34:49.947 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-10T20:34:50.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient: get_monmap_and_config 2024-09-10T20:34:50.070 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient: build_initial_monmap 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient: monmap: 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:50.070340+0000 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:50.070340+0000 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:50.071 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:50.072 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:50.072 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:50.072 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.073 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 1 Processor -- start 2024-09-10T20:34:50.076 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 1 -- start start 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient: init 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0ef81364f0 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ef81360a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f0ef80a1990 0x7f0ef813eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ef80a1990 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef80a27a0 con 0x7f0ef81364f0 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ef80a2a80 con 0x7f0ef81360a0 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.077 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f0ef80a2550 con 0x7f0ef80a1990 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0f002fa640 10 monclient(hunting): authenticate will time out at 1450.210000s 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe096640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f0ef81364f0 0x7f0ef81368d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48326/0 (socket says 172.21.15.153:48326) 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe096640 1 -- 172.21.15.153:0/2332788240 learned_addr learned my addr 172.21.15.153:0/2332788240 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 1 --2- 172.21.15.153:0/2332788240 >> v2:172.21.15.153:3300/0 conn(0x7f0ef80a1990 0x7f0ef813eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient(hunting): get_auth_request con 0x7f0ef80a1990 auth_method 0 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.078 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:50.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:50.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:50.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient(hunting): handle_auth_done global_id 34368 payload 274 2024-09-10T20:34:50.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 1 -- 172.21.15.153:0/2332788240 >> v1:172.21.15.176:6789/0 conn(0x7f0ef81360a0 legacy=0x7f0ef813b420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:50.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 1 -- 172.21.15.153:0/2332788240 >> v1:172.21.15.153:6789/0 conn(0x7f0ef81364f0 legacy=0x7f0ef81368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:50.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient: _finish_hunting 0 2024-09-10T20:34:50.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 1 monclient: found mon.noname-a 2024-09-10T20:34:50.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:50.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.080 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.069+0000 7f0efe897640 1 -- 172.21.15.153:0/2332788240 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef80a2d60 con 0x7f0ef80a1990 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efe897640 1 -- 172.21.15.153:0/2332788240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 msgr2=0x7f0ef813eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efe897640 1 --2- 172.21.15.153:0/2332788240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 0x7f0ef813eb50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009d90 tx=0x7f0ef001c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea89355381f6e759 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 1 -- 172.21.15.153:0/2332788240 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef00b2070 con 0x7f0ef80a1990 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 10 monclient: dump: 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.088 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 10 monclient: _finish_auth 0 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 10 monclient: _check_auth_tickets 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 1 -- 172.21.15.153:0/2332788240 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ef00a5770 con 0x7f0ef80a1990 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 5 monclient: authenticate success, global_id 34368 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 1 -- 172.21.15.153:0/2332788240 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef00a5b40 con 0x7f0ef80a1990 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 4 set_mon_vals no callback set 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.089 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0efd094640 10 monclient: dump: 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 10 monclient: shutdown 2024-09-10T20:34:50.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 1 -- 172.21.15.153:0/2332788240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 msgr2=0x7f0ef813eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 1 --2- 172.21.15.153:0/2332788240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 0x7f0ef813eb50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f0ef0009d90 tx=0x7f0ef001c830 comp rx=0 tx=0).stop 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 1 -- 172.21.15.153:0/2332788240 shutdown_connections 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 1 --2- 172.21.15.153:0/2332788240 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 0x7f0ef813eb50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 1 -- 172.21.15.153:0/2332788240 >> 172.21.15.153:0/2332788240 conn(0x7f0ef80a71b0 msgr2=0x7f0ef80a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.073+0000 7f0f002fa640 1 -- 172.21.15.153:0/2332788240 shutdown_connections 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 -- 172.21.15.153:0/2332788240 wait complete. 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient: build_initial_monmap 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient: monmap: 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:50.091 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 librados: starting msgr at 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 librados: starting objecter 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 Processor -- start 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 -- start start 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 librados: setting wanted keys 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 librados: calling monclient init 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient: init 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:50.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 0x7f0ef8139890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient(hunting): picked mon.noname-a con 0x7f0ef80a1990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef81360a0 0x7f0ef81370a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient(hunting): picked mon.noname-b con 0x7f0ef81360a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef81364f0 0x7f0ef81375e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient(hunting): picked mon.noname-c con 0x7f0ef81364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef813fd70 con 0x7f0ef80a1990 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.093 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef813fbf0 con 0x7f0ef81360a0 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f0ef813fef0 con 0x7f0ef81364f0 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0f002fa640 10 monclient(hunting): authenticate will time out at 1450.217428s 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef81364f0 0x7f0ef81375e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 0x7f0ef8139890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef81364f0 0x7f0ef81375e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:60418/0 (socket says 172.21.15.153:60418) 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 1 -- 172.21.15.153:0/2934272961 learned_addr learned my addr 172.21.15.153:0/2934272961 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 10 monclient(hunting): get_auth_request con 0x7f0ef81364f0 auth_method 0 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.094 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient(hunting): get_auth_request con 0x7f0ef80a1990 auth_method 0 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efd895640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef81360a0 0x7f0ef81370a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efd895640 10 monclient(hunting): get_auth_request con 0x7f0ef81360a0 auth_method 0 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efd895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.095 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efd895640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efd895640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient(hunting): handle_auth_done global_id 34371 payload 824 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef81364f0 msgr2=0x7f0ef81375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef81364f0 0x7f0ef81375e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef81360a0 msgr2=0x7f0ef81370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef81360a0 0x7f0ef81370a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient: _finish_hunting 0 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 1 monclient: found mon.noname-a 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.096 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe096640 1 -- 172.21.15.153:0/2934272961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ef80a6690 con 0x7f0ef80a1990 2024-09-10T20:34:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.077+0000 7f0efe897640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f0ef81364f0 0x7f0ef81375e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efe096640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 msgr2=0x7f0ef8139890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efe096640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 0x7f0ef8139890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009450 tx=0x7f0ee8009910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 client.?.objecter ms_handle_connect 0x7f0ef80a1990 2024-09-10T20:34:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8018070 con 0x7f0ef80a1990 2024-09-10T20:34:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:50.097 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 monclient: dump: 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 monclient: _finish_auth 0 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 monclient: _check_auth_tickets 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:50.098 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ee80062e0 con 0x7f0ef80a1990 2024-09-10T20:34:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ee80047d0 con 0x7f0ef80a1990 2024-09-10T20:34:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 monclient: dump: 2024-09-10T20:34:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.099 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 5 monclient: authenticate success, global_id 34371 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 10 monclient: _renew_subs 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ef80a6ba0 con 0x7f0ef80a1990 2024-09-10T20:34:50.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeffff640 4 set_mon_vals no callback set 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 10 client.34371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 10 monclient: _renew_subs 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ef8139dd0 con 0x7f0ef80a1990 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 1 librados: init done 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef8137b20 0x7f0ef806f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef809e0a0 con 0x7f0ef8137b20 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f0ee8004a70 con 0x7f0ef80a1990 2024-09-10T20:34:50.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef8137b20 0x7f0ef806f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 10 monclient: get_auth_request con 0x7f0ef8137b20 auth_method 0 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 1 --2- 172.21.15.153:0/2934272961 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0edc03cef0 0x7f0edc03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f0ee804ee60 con 0x7f0ef80a1990 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 10 client.34371.objecter ms_dispatch 0x7f0ef8000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 3 client.34371.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 3 client.34371.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:50.102 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0eeeffd640 20 client.34371.objecter dump_active .. 0 homeless 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 10 monclient(hunting): handle_auth_done global_id 20200 payload 824 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efe897640 1 --2- 172.21.15.153:0/2934272961 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0edc03cef0 0x7f0edc03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efd895640 1 -- 172.21.15.153:0/2934272961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0ef8137f00 con 0x7f0ef8137b20 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.081+0000 7f0efe897640 10 monclient: get_auth_request con 0x7f0edc03cef0 auth_method 0 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 10 client.34371.objecter ms_handle_connect 0x7f0ef8137b20 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 10 client.34371.objecter resend_mon_ops 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0efd895640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef8137b20 msgr2=0x7f0ef806f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0efd895640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef8137b20 0x7f0ef806f830 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400e450 tx=0x7f0ef400e910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef4007450 con 0x7f0ef8137b20 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 10 monclient: dump: 2024-09-10T20:34:50.103 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.104 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.104 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.104 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.104 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.104 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.104 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.104 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.105 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.105 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 10 client.34371.objecter ms_handle_connect 0x7f0edc03cef0 2024-09-10T20:34:50.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0efe897640 1 --2- 172.21.15.153:0/2934272961 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0edc03cef0 0x7f0edc03f3b0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f0ef001c870 tx=0x7f0ef00a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:50.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0ef4007450 con 0x7f0ef8137b20 2024-09-10T20:34:50.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:50.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:50.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef8137b20 msgr2=0x7f0ef806f830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef8137b20 0x7f0ef806f830 secure :-1 s=READY pgs=5763 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400e450 tx=0x7f0ef400e910 comp rx=0 tx=0).stop 2024-09-10T20:34:50.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef8137b20 msgr2=0x7f0ef806f830 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:34:50.106 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.085+0000 7f0eeeffd640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef8137b20 0x7f0ef806f830 unknown :-1 s=CLOSED pgs=5763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.089+0000 7f0f002fa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-10T20:34:50.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.089+0000 7f0f002fa640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef806fd70 0x7f0ef8070180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.107 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.089+0000 7f0f002fa640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.089+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f0ef813aaf0 con 0x7f0ef806fd70 2024-09-10T20:34:50.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef806fd70 0x7f0ef8070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 10 monclient: get_auth_request con 0x7f0ef806fd70 auth_method 0 2024-09-10T20:34:50.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.108 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 10 monclient(hunting): handle_auth_done global_id 20203 payload 824 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 1 -- 172.21.15.153:0/2934272961 --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0ef81368d0 con 0x7f0ef806fd70 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef806fd70 msgr2=0x7f0ef8070180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 10 client.34371.objecter ms_handle_connect 0x7f0ef806fd70 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 10 client.34371.objecter resend_mon_ops 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0efd895640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef806fd70 0x7f0ef8070180 secure :-1 s=READY pgs=5764 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400e450 tx=0x7f0ef4003770 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ef401eb70 con 0x7f0ef806fd70 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 10 monclient: dump: 2024-09-10T20:34:50.109 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 <== mon.1 v2:172.21.15.176:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4887 (secure 0 0 0) 0x7f0ef401eb70 con 0x7f0ef806fd70 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:34:50.110 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef806fd70 msgr2=0x7f0ef8070180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef806fd70 0x7f0ef8070180 secure :-1 s=READY pgs=5764 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400e450 tx=0x7f0ef4003770 comp rx=0 tx=0).stop 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef806fd70 msgr2=0x7f0ef8070180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.093+0000 7f0eeeffd640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f0ef806fd70 0x7f0ef8070180 secure :-1 s=CLOSED pgs=5764 cs=0 l=1 rev1=1 crypto rx=0x7f0ef400e450 tx=0x7f0ef4003770 comp rx=0 tx=0).stop 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stdout: "state": "peon", 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stdout: "election_epoch": 38, 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stdout: "uptime": 536736, 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum": [ 2024-09-10T20:34:50.111 INFO:teuthology.orchestra.run.smithi153.stdout: 0, 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: 1, 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: 2 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_age": 3, 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "required_con": "2450521708007866372", 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "required_mon": [ 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:34:50.112 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_con": "4540719139924082687", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "quorum_mon": [ 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:34:50.113 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: "outside_quorum": [], 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: "extra_probe_peers": [], 2024-09-10T20:34:50.114 INFO:teuthology.orchestra.run.smithi153.stdout: "sync_provider": [], 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "monmap": { 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "epoch": 1, 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "fsid": "8126fdd8-2c77-4da0-ba28-d5fdc347eec8", 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "modified": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "created": "2024-09-10T20:25:26.988641Z", 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release": 19, 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "min_mon_release_name": "squid", 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "election_strategy": 1, 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "disallowed_leaders: ": "", 2024-09-10T20:34:50.115 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false, 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "tiebreaker_mon": "", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "removed_ranks: ": "", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "features": { 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "persistent": [ 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "kraken", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "luminous", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "mimic", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "osdmap-prune", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "nautilus", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "octopus", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "pacific", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "elector-pinging", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "quincy", 2024-09-10T20:34:50.116 INFO:teuthology.orchestra.run.smithi153.stdout: "reef", 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: "squid" 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: "optional": [] 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: "mons": [ 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 0, 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "a", 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.117 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3300", 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789", 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6789/0", 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6789/0", 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:34:50.118 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 1, 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "b", 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:3300", 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:50.119 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789", 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.176:6789/0", 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.176:6789/0", 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:34:50.120 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:34:50.121 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:34:50.121 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.121 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.121 INFO:teuthology.orchestra.run.smithi153.stdout: "rank": 2, 2024-09-10T20:34:50.121 INFO:teuthology.orchestra.run.smithi153.stdout: "name": "c", 2024-09-10T20:34:50.121 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addrs": { 2024-09-10T20:34:50.122 INFO:teuthology.orchestra.run.smithi153.stdout: "addrvec": [ 2024-09-10T20:34:50.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.122 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v2", 2024-09-10T20:34:50.122 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:3301", 2024-09-10T20:34:50.122 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:50.122 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.122 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.122 INFO:teuthology.orchestra.run.smithi153.stdout: "type": "v1", 2024-09-10T20:34:50.122 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790", 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: "nonce": 0 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: "addr": "172.21.15.153:6790/0", 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: "public_addr": "172.21.15.153:6790/0", 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: "priority": 0, 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: "weight": 0, 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: "crush_location": "{}" 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:50.123 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:50.124 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.124 INFO:teuthology.orchestra.run.smithi153.stdout: "feature_map": { 2024-09-10T20:34:50.124 INFO:teuthology.orchestra.run.smithi153.stdout: "mon": [ 2024-09-10T20:34:50.124 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.124 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:34:50.124 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:34:50.124 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:34:50.124 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: "osd": [ 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 4 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: "client": [ 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 5 2024-09-10T20:34:50.125 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: ], 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: "mgr": [ 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: { 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: "features": "0x3f03dffffffdffff", 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: "release": "squid", 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: "num": 1 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: } 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: ] 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: }, 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout: "stretch_mode": false 2024-09-10T20:34:50.126 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:34:50.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 10 librados: watch_flush enter 2024-09-10T20:34:50.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 10 librados: watch_flush exit 2024-09-10T20:34:50.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 20 client.34371.objecter shutdown clearing up homeless session... 2024-09-10T20:34:50.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 10 client.34371.objecter successfully canceled tick 2024-09-10T20:34:50.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0edc03cef0 msgr2=0x7f0edc03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 1 --2- 172.21.15.153:0/2934272961 >> v2:172.21.15.176:6812/3003473398 conn(0x7f0edc03cef0 0x7f0edc03f3b0 secure :-1 s=READY pgs=3369 cs=0 l=1 rev1=1 crypto rx=0x7f0ef001c870 tx=0x7f0ef00a7070 comp rx=0 tx=0).stop 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 10 monclient: shutdown 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 msgr2=0x7f0ef8139890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 1 --2- 172.21.15.153:0/2934272961 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f0ef80a1990 0x7f0ef8139890 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009450 tx=0x7f0ee8009910 comp rx=0 tx=0).stop 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0efe897640 1 -- 172.21.15.153:0/2934272961 reap_dead start 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0efe897640 1 -- 172.21.15.153:0/2934272961 reap_dead start 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 shutdown_connections 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 >> 172.21.15.153:0/2934272961 conn(0x7f0ef80a71b0 msgr2=0x7f0ef809d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 shutdown_connections 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 1 -- 172.21.15.153:0/2934272961 wait complete. 2024-09-10T20:34:50.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.097+0000 7f0f002fa640 1 librados: shutdown 2024-09-10T20:34:50.130 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-10T20:34:50.130 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-10T20:34:50.130 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-10T20:34:50.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.233+0000 7fd66c0c0640 10 monclient: get_monmap_and_config 2024-09-10T20:34:50.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.233+0000 7fd66c0c0640 10 monclient: build_initial_monmap 2024-09-10T20:34:50.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.233+0000 7fd66c0c0640 10 monclient: monmap: 2024-09-10T20:34:50.237 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:50.237354+0000 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:50.237354+0000 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:50.238 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:50.239 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 1 Processor -- start 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 1 -- start start 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 10 monclient: init 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 10 monclient: _add_conns ranks=[0,1,3,4,2] 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd6640a5ab0 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd6640a65a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd66413f390 0x7fd664141780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 10 monclient(hunting): picked mon.noname-a con 0x7fd66413f390 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd664070780 con 0x7fd6640a5ab0 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6640709c0 con 0x7fd6640a65a0 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fd6640a3010 con 0x7fd66413f390 2024-09-10T20:34:50.241 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66c0c0640 10 monclient(hunting): authenticate will time out at 1450.376676s 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fd66413f390 0x7fd664141780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd669e5c640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fd6640a5ab0 0x7fd6640a5e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:48334/0 (socket says 172.21.15.153:48334) 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd669e5c640 1 -- 172.21.15.153:0/3426884480 learned_addr learned my addr 172.21.15.153:0/3426884480 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient(hunting): get_auth_request con 0x7fd66413f390 auth_method 0 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient(hunting): handle_auth_done global_id 34374 payload 274 2024-09-10T20:34:50.242 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 1 -- 172.21.15.153:0/3426884480 >> v1:172.21.15.176:6789/0 conn(0x7fd6640a65a0 legacy=0x7fd66413dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:50.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 1 -- 172.21.15.153:0/3426884480 >> v1:172.21.15.153:6789/0 conn(0x7fd6640a5ab0 legacy=0x7fd6640a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:50.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient: _finish_hunting 0 2024-09-10T20:34:50.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 1 monclient: found mon.noname-a 2024-09-10T20:34:50.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:50.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.237+0000 7fd66a65d640 1 -- 172.21.15.153:0/3426884480 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6640a26e0 con 0x7fd66413f390 2024-09-10T20:34:50.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66a65d640 1 -- 172.21.15.153:0/3426884480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66413f390 msgr2=0x7fd664141780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66a65d640 1 --2- 172.21.15.153:0/3426884480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66413f390 0x7fd664141780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd65c009d90 tx=0x7fd65c01c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86f78dd8b1272011 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 1 -- 172.21.15.153:0/3426884480 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0b2070 con 0x7fd66413f390 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 10 monclient: dump: 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.248 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 10 monclient: _finish_auth 0 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 10 monclient: _check_auth_tickets 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 1 -- 172.21.15.153:0/3426884480 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd65c0a5860 con 0x7fd66413f390 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 1 -- 172.21.15.153:0/3426884480 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd65c0a5c30 con 0x7fd66413f390 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 5 monclient: authenticate success, global_id 34374 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:50.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 4 set_mon_vals no callback set 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd668e5a640 10 monclient: dump: 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.250 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 10 monclient: shutdown 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3426884480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66413f390 msgr2=0x7fd664141780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 --2- 172.21.15.153:0/3426884480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66413f390 0x7fd664141780 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fd65c009d90 tx=0x7fd65c01c830 comp rx=0 tx=0).stop 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3426884480 shutdown_connections 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 --2- 172.21.15.153:0/3426884480 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd66413f390 0x7fd664141780 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3426884480 >> 172.21.15.153:0/3426884480 conn(0x7fd6640aba00 msgr2=0x7fd6640a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3426884480 shutdown_connections 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3426884480 wait complete. 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 10 monclient: build_initial_monmap 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 10 monclient: monmap: 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:50.251 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 librados: starting msgr at 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 librados: starting objecter 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 Processor -- start 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 -- start start 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 librados: setting wanted keys 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 1 librados: calling monclient init 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.241+0000 7fd66c0c0640 10 monclient: init 2024-09-10T20:34:50.252 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6640a5ab0 0x7fd664137b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient(hunting): picked mon.noname-c con 0x7fd6640a5ab0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6640a65a0 0x7fd6641380d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient(hunting): picked mon.noname-a con 0x7fd6640a65a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd66413f390 0x7fd66413b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient(hunting): picked mon.noname-b con 0x7fd66413f390 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fd6640a3b70 con 0x7fd6640a65a0 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fd6641444e0 con 0x7fd66413f390 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:50.253 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fd6640a22a0 con 0x7fd6640a5ab0 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient(hunting): authenticate will time out at 1450.384270s 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6640a65a0 0x7fd6641380d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd669e5c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6640a5ab0 0x7fd664137b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6640a65a0 0x7fd6641380d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:47424/0 (socket says 172.21.15.153:47424) 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd669e5c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6640a5ab0 0x7fd664137b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:60432/0 (socket says 172.21.15.153:60432) 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 -- 172.21.15.153:0/3959243763 learned_addr learned my addr 172.21.15.153:0/3959243763 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient(hunting): get_auth_request con 0x7fd6640a65a0 auth_method 0 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.254 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd669e5c640 10 monclient(hunting): get_auth_request con 0x7fd6640a5ab0 auth_method 0 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd669e5c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd669e5c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd669e5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd669e5c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd669e5c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd669e5c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66a65d640 1 --2- 172.21.15.153:0/3959243763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd66413f390 0x7fd66413b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66a65d640 10 monclient(hunting): get_auth_request con 0x7fd66413f390 auth_method 0 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66a65d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:50.255 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66a65d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66a65d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient(hunting): handle_auth_done global_id 34377 payload 824 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 -- 172.21.15.153:0/3959243763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6640a5ab0 msgr2=0x7fd664137b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 --2- 172.21.15.153:0/3959243763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6640a5ab0 0x7fd664137b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 -- 172.21.15.153:0/3959243763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd66413f390 msgr2=0x7fd66413b220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 --2- 172.21.15.153:0/3959243763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd66413f390 0x7fd66413b220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient: _finish_hunting 0 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 monclient: found mon.noname-a 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.256 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 -- 172.21.15.153:0/3959243763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd664121a80 con 0x7fd6640a65a0 2024-09-10T20:34:50.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 -- 172.21.15.153:0/3959243763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6640a65a0 msgr2=0x7fd6641380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66965b640 1 --2- 172.21.15.153:0/3959243763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6640a65a0 0x7fd6641380d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd66000c450 tx=0x7fd66000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 client.?.objecter ms_handle_connect 0x7fd6640a65a0 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 1 -- 172.21.15.153:0/3959243763 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660012a40 con 0x7fd6640a65a0 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 monclient: dump: 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.259 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 monclient: _finish_auth 0 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 1 -- 172.21.15.153:0/3959243763 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd660017070 con 0x7fd6640a65a0 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 1 -- 172.21.15.153:0/3959243763 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd660012e40 con 0x7fd6640a65a0 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 5 monclient: authenticate success, global_id 34377 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient: _renew_subs 2024-09-10T20:34:50.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd66413ea80 con 0x7fd6640a65a0 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65bfff640 4 set_mon_vals no callback set 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd65a7fc640 10 monclient: dump: 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:50.261 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 client.34377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient: _renew_subs 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd66413dbf0 con 0x7fd6640a65a0 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.245+0000 7fd66c0c0640 1 librados: init done 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd66c0c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd66c0c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd66c0c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6640a5e90 con 0x7fd6640a65a0 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd65a7fc640 1 -- 172.21.15.153:0/3959243763 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fd66001c460 con 0x7fd6640a65a0 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd65a7fc640 1 --2- 172.21.15.153:0/3959243763 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd64803cf20 0x7fd64803f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd65a7fc640 1 -- 172.21.15.153:0/3959243763 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7fd66004f300 con 0x7fd6640a65a0 2024-09-10T20:34:50.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd65a7fc640 10 client.34377.objecter ms_dispatch 0x7fd664000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd65a7fc640 3 client.34377.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd65a7fc640 3 client.34377.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd65a7fc640 20 client.34377.objecter dump_active .. 0 homeless 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd669e5c640 1 --2- 172.21.15.153:0/3959243763 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd64803cf20 0x7fd64803f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd669e5c640 10 monclient: get_auth_request con 0x7fd64803cf20 auth_method 0 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd669e5c640 1 --2- 172.21.15.153:0/3959243763 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd64803cf20 0x7fd64803f3e0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fd654007ca0 tx=0x7fd654007bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.249+0000 7fd65a7fc640 10 client.34377.objecter ms_handle_connect 0x7fd64803cf20 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.253+0000 7fd65a7fc640 1 -- 172.21.15.153:0/3959243763 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fd66001ac60 con 0x7fd6640a65a0 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.253+0000 7fd65a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:50.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.253+0000 7fd65a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:50.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.465+0000 7fd66c0c0640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-10T20:34:50.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.465+0000 7fd66c0c0640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-10T20:34:50.469 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.465+0000 7fd66c0c0640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:50.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.465+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fd6640a6be0 con 0x7fd6640a65a0 2024-09-10T20:34:50.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.469+0000 7fd65a7fc640 1 -- 172.21.15.153:0/3959243763 <== mon.0 v2:172.21.15.153:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fd66001c930 con 0x7fd6640a65a0 2024-09-10T20:34:50.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.469+0000 7fd65a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-10T20:34:50.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.469+0000 7fd65a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:34:50.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.469+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.469+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=100120495,config=3541143443,health=2099252208,kv=3615289851,logm=1963871015}) 2024-09-10T20:34:50.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.469+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=13,config=2,health=12,kv=10,logm=63} crc {auth=1958780896,config=3541143443,health=2099252208,kv=2875136694,logm=2107196261}) 2024-09-10T20:34:50.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.469+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.469+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=10,config=2,health=12,kv=11,logm=65} crc {auth=100120495,config=3541143443,health=2099252208,kv=3615289851,logm=1963871015}) 2024-09-10T20:34:50.473 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.469+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=1,health=12,kv=10,logm=65} crc {auth=523928794,config=555582061,health=2392403904,kv=3615289851,logm=1927319710}) 2024-09-10T20:34:50.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 10 librados: watch_flush enter 2024-09-10T20:34:50.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 10 librados: watch_flush exit 2024-09-10T20:34:50.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 20 client.34377.objecter shutdown clearing up homeless session... 2024-09-10T20:34:50.474 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 10 client.34377.objecter successfully canceled tick 2024-09-10T20:34:50.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249513649}) 2024-09-10T20:34:50.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4002455578}) 2024-09-10T20:34:50.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2249513649}) 2024-09-10T20:34:50.475 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2115443566}) 2024-09-10T20:34:50.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd64803cf20 msgr2=0x7fd64803f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 --2- 172.21.15.153:0/3959243763 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd64803cf20 0x7fd64803f3e0 secure :-1 s=READY pgs=3370 cs=0 l=1 rev1=1 crypto rx=0x7fd654007ca0 tx=0x7fd654007bf0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 10 monclient: shutdown 2024-09-10T20:34:50.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:50.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6640a65a0 msgr2=0x7fd6641380d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:50.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 --2- 172.21.15.153:0/3959243763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6640a65a0 0x7fd6641380d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd66000c450 tx=0x7fd66000c910 comp rx=0 tx=0).stop 2024-09-10T20:34:50.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 shutdown_connections 2024-09-10T20:34:50.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 --2- 172.21.15.153:0/3959243763 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fd66413f390 0x7fd66413b220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 --2- 172.21.15.153:0/3959243763 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fd6640a65a0 0x7fd6641380d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 --2- 172.21.15.153:0/3959243763 >> v2:172.21.15.176:6812/3003473398 conn(0x7fd64803cf20 0x7fd64803f3e0 unknown :-1 s=CLOSED pgs=3370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 --2- 172.21.15.153:0/3959243763 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fd6640a5ab0 0x7fd664137b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:50.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 >> 172.21.15.153:0/3959243763 conn(0x7fd6640aba00 msgr2=0x7fd6640a93e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:50.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 shutdown_connections 2024-09-10T20:34:50.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 -- 172.21.15.153:0/3959243763 wait complete. 2024-09-10T20:34:50.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fd66c0c0640 1 librados: shutdown 2024-09-10T20:34:50.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2493540353}) 2024-09-10T20:34:50.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3085413425}) 2024-09-10T20:34:50.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.479 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2493540353}) 2024-09-10T20:34:50.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.473+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3023619964}) 2024-09-10T20:34:50.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2181931329}) 2024-09-10T20:34:50.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4157680151}) 2024-09-10T20:34:50.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2181931329}) 2024-09-10T20:34:50.480 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=261853457}) 2024-09-10T20:34:50.482 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802246934}) 2024-09-10T20:34:50.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3035139226}) 2024-09-10T20:34:50.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802246934}) 2024-09-10T20:34:50.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=819207208}) 2024-09-10T20:34:50.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.483 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183891427}) 2024-09-10T20:34:50.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.477+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=707431248}) 2024-09-10T20:34:50.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1183891427}) 2024-09-10T20:34:50.484 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4011102290}) 2024-09-10T20:34:50.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272450673}) 2024-09-10T20:34:50.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4234246889}) 2024-09-10T20:34:50.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272450673}) 2024-09-10T20:34:50.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1848281548}) 2024-09-10T20:34:50.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.490 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717090483}) 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2570599325}) 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=717090483}) 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.481+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=958165978}) 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2950742822}) 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147832223}) 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2950742822}) 2024-09-10T20:34:50.491 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=954365148}) 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3422823205}) 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=136790690}) 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3422823205}) 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.485+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=923897432}) 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230294009}) 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3992679516}) 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230294009}) 2024-09-10T20:34:50.492 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=536703353}) 2024-09-10T20:34:50.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297881278}) 2024-09-10T20:34:50.493 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1920436643}) 2024-09-10T20:34:50.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2297881278}) 2024-09-10T20:34:50.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3859987819}) 2024-09-10T20:34:50.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289817220}) 2024-09-10T20:34:50.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684333322}) 2024-09-10T20:34:50.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3289817220}) 2024-09-10T20:34:50.494 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.489+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=425883654}) 2024-09-10T20:34:50.495 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-10T20:34:50.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.496 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652356816}) 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2236765686}) 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1652356816}) 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=522257983}) 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778868266}) 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2948548610}) 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1778868266}) 2024-09-10T20:34:50.497 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.493+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3821718115}) 2024-09-10T20:34:50.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022934768}) 2024-09-10T20:34:50.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3057158548}) 2024-09-10T20:34:50.498 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2022934768}) 2024-09-10T20:34:50.499 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=961270590}) 2024-09-10T20:34:50.500 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.500 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386793621}) 2024-09-10T20:34:50.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=337414970}) 2024-09-10T20:34:50.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3386793621}) 2024-09-10T20:34:50.501 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.497+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2719348813}) 2024-09-10T20:34:50.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554885387}) 2024-09-10T20:34:50.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=557129546}) 2024-09-10T20:34:50.502 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=554885387}) 2024-09-10T20:34:50.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=518034949}) 2024-09-10T20:34:50.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192637529}) 2024-09-10T20:34:50.503 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3923133319}) 2024-09-10T20:34:50.504 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.504 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=192637529}) 2024-09-10T20:34:50.504 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2192366222}) 2024-09-10T20:34:50.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130159340}) 2024-09-10T20:34:50.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2925710218}) 2024-09-10T20:34:50.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2130159340}) 2024-09-10T20:34:50.505 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.501+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2196273199}) 2024-09-10T20:34:50.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632983203}) 2024-09-10T20:34:50.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4210796667}) 2024-09-10T20:34:50.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=632983203}) 2024-09-10T20:34:50.507 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3258937968}) 2024-09-10T20:34:50.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534204658}) 2024-09-10T20:34:50.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2590875375}) 2024-09-10T20:34:50.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1534204658}) 2024-09-10T20:34:50.509 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.505+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1586097319}) 2024-09-10T20:34:50.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.510 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313435513}) 2024-09-10T20:34:50.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2592482937}) 2024-09-10T20:34:50.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2313435513}) 2024-09-10T20:34:50.511 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3873602998}) 2024-09-10T20:34:50.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075150023}) 2024-09-10T20:34:50.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1194188663}) 2024-09-10T20:34:50.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4075150023}) 2024-09-10T20:34:50.512 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2139260614}) 2024-09-10T20:34:50.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.513 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032248346}) 2024-09-10T20:34:50.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254523661}) 2024-09-10T20:34:50.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2032248346}) 2024-09-10T20:34:50.514 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.509+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1333849139}) 2024-09-10T20:34:50.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.533+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.533+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3795764995}) 2024-09-10T20:34:50.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.533+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3080974548}) 2024-09-10T20:34:50.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.533+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.533+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3795764995}) 2024-09-10T20:34:50.536 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.533+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2720835200}) 2024-09-10T20:34:50.550 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.545+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.550 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.545+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026620765}) 2024-09-10T20:34:50.550 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.549+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3319744711}) 2024-09-10T20:34:50.550 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.549+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.550 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.549+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1026620765}) 2024-09-10T20:34:50.550 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.549+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2021569751}) 2024-09-10T20:34:50.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.897+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.897+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562037603}) 2024-09-10T20:34:50.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.897+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3313229948}) 2024-09-10T20:34:50.902 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.897+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.901+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1562037603}) 2024-09-10T20:34:50.903 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.901+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3205379003}) 2024-09-10T20:34:50.904 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.901+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.904 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.901+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639947422}) 2024-09-10T20:34:50.904 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.901+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2911563460}) 2024-09-10T20:34:50.904 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.901+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.904 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.901+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1639947422}) 2024-09-10T20:34:50.905 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.901+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1557681925}) 2024-09-10T20:34:50.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.906 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221280630}) 2024-09-10T20:34:50.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3142448438}) 2024-09-10T20:34:50.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3221280630}) 2024-09-10T20:34:50.907 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1674020799}) 2024-09-10T20:34:50.908 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.908 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938090442}) 2024-09-10T20:34:50.909 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3982629011}) 2024-09-10T20:34:50.909 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.909 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2938090442}) 2024-09-10T20:34:50.909 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.905+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2609650786}) 2024-09-10T20:34:50.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2459354272}) 2024-09-10T20:34:50.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1409515409}) 2024-09-10T20:34:50.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2459354272}) 2024-09-10T20:34:50.910 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1916251260}) 2024-09-10T20:34:50.912 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=1} crc {logm=2393604265,mdsmap=3397941211,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=3178573270,mon_config_key=2240418744}) 2024-09-10T20:34:50.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=76,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5} crc {logm=493614707,mdsmap=3140514009,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=3312985760,mon_config_key=895648768}) 2024-09-10T20:34:50.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=1} crc {logm=2393604265,mdsmap=3397941211,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=3178573270,mon_config_key=2240418744}) 2024-09-10T20:34:50.913 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.909+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=78,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=1} crc {logm=1917549427,mdsmap=3397941211,mgr=3323764283,mgr_command_descs=2308730518,mgr_metadata=3232925821,mgrstat=3178573270,mon_config_key=2240418744}) 2024-09-10T20:34:50.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.913+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.913+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=58} crc {mon_config_key=1531656394,monmap=999369809,nvmeofgw=598337686,osd_metadata=753924036,osd_pg_creating=2093958268,osd_snap=2216181629,osdmap=1515243875}) 2024-09-10T20:34:50.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.913+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=58} crc {mon_config_key=3282649592,monmap=999369809,nvmeofgw=598337686,osd_metadata=753924036,osd_pg_creating=2093958268,osd_snap=2216181629,osdmap=122464775}) 2024-09-10T20:34:50.915 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.913+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.913+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=58} crc {mon_config_key=1531656394,monmap=999369809,nvmeofgw=598337686,osd_metadata=753924036,osd_pg_creating=2093958268,osd_snap=2216181629,osdmap=1515243875}) 2024-09-10T20:34:50.916 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.913+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=13,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=17,osdmap=58} crc {mon_config_key=1534211818,monmap=3140514009,nvmeofgw=598337686,osd_metadata=753924036,osd_pg_creating=2093958268,osd_snap=2216181629,osdmap=1165056389}) 2024-09-10T20:34:50.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.917+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.917+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4195870923}) 2024-09-10T20:34:50.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.917+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1015541639}) 2024-09-10T20:34:50.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.917+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.918 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.917+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4195870923}) 2024-09-10T20:34:50.919 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.917+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=258036324}) 2024-09-10T20:34:50.930 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.929+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.930 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.929+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=4136971411}) 2024-09-10T20:34:50.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.929+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=19} crc {osdmap=1055590322}) 2024-09-10T20:34:50.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.929+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-10T20:34:50.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.929+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=19} crc {osdmap=4136971411}) 2024-09-10T20:34:50.931 INFO:tasks.ceph.mon.a.smithi153.stderr:2024-09-10T20:34:50.929+0000 7fdd54a19640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=17} crc {osdmap=754497579}) 2024-09-10T20:34:51.497 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-10T20:34:51.497 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-10T20:34:51.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient: get_monmap_and_config 2024-09-10T20:34:51.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient: build_initial_monmap 2024-09-10T20:34:51.627 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient: monmap: 2024-09-10T20:34:51.627 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:51.627 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:51.627 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:51.627154+0000 2024-09-10T20:34:51.628 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:51.627154+0000 2024-09-10T20:34:51.628 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:51.628 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:51.628 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:51.628 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:51.628 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:51.628 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:51.628 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:51.628 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 1 Processor -- start 2024-09-10T20:34:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 1 -- start start 2024-09-10T20:34:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient: init 2024-09-10T20:34:51.630 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a41a0 0x7f8a140a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient(hunting): picked mon.noname-b con 0x7f8a140a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8a140a4580 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8a140a49a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a140a67f0 con 0x7f8a140a49a0 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a140a6a20 con 0x7f8a140a4580 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8a140a2570 con 0x7f8a140a41a0 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a1a61e640 10 monclient(hunting): authenticate will time out at 1451.767552s 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a18bbb640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f8a140a49a0 0x7f8a140a60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60692/0 (socket says 172.21.15.153:60692) 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a18bbb640 1 -- 172.21.15.153:0/1226331275 learned_addr learned my addr 172.21.15.153:0/1226331275 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:51.631 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a13fff640 1 --2- 172.21.15.153:0/1226331275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a41a0 0x7f8a140a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a13fff640 10 monclient(hunting): get_auth_request con 0x7f8a140a41a0 auth_method 0 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a13fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.625+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.629+0000 7f8a13fff640 10 monclient(hunting): handle_auth_done global_id 54113 payload 274 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.629+0000 7f8a13fff640 1 -- 172.21.15.153:0/1226331275 >> v1:172.21.15.176:6789/0 conn(0x7f8a140a4580 legacy=0x7f8a140a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.629+0000 7f8a13fff640 1 -- 172.21.15.153:0/1226331275 >> v1:172.21.15.153:6789/0 conn(0x7f8a140a49a0 legacy=0x7f8a140a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.629+0000 7f8a13fff640 10 monclient: _finish_hunting 0 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.629+0000 7f8a13fff640 1 monclient: found mon.noname-b 2024-09-10T20:34:51.632 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.629+0000 7f8a13fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.629+0000 7f8a13fff640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:51.633 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.629+0000 7f8a13fff640 1 -- 172.21.15.153:0/1226331275 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a140a3c80 con 0x7f8a140a41a0 2024-09-10T20:34:51.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 1 -- 172.21.15.153:0/1226331275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a41a0 msgr2=0x7f8a140a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:51.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 1 --2- 172.21.15.153:0/1226331275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a41a0 0x7f8a140a3740 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8a040097f0 tx=0x7f8a0401c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9dfa6250caeb7ba server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:51.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 1 -- 172.21.15.153:0/1226331275 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a040b0070 con 0x7f8a140a41a0 2024-09-10T20:34:51.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:51.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:51.634 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 10 monclient: dump: 2024-09-10T20:34:51.634 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 10 monclient: _finish_auth 0 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 10 monclient: _check_auth_tickets 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 1 -- 172.21.15.153:0/1226331275 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a040ab600 con 0x7f8a140a41a0 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 5 monclient: authenticate success, global_id 54113 2024-09-10T20:34:51.635 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 1 -- 172.21.15.153:0/1226331275 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a040ab9d0 con 0x7f8a140a41a0 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 4 set_mon_vals no callback set 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a137fe640 10 monclient: dump: 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:51.636 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:51.637 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:51.637 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:51.637 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:51.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient: shutdown 2024-09-10T20:34:51.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:51.641 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 -- 172.21.15.153:0/1226331275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a41a0 msgr2=0x7f8a140a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 --2- 172.21.15.153:0/1226331275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a41a0 0x7f8a140a3740 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8a040097f0 tx=0x7f8a0401c3a0 comp rx=0 tx=0).stop 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 -- 172.21.15.153:0/1226331275 shutdown_connections 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 --2- 172.21.15.153:0/1226331275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a41a0 0x7f8a140a3740 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 -- 172.21.15.153:0/1226331275 >> 172.21.15.153:0/1226331275 conn(0x7f8a140ab7e0 msgr2=0x7f8a140a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 -- 172.21.15.153:0/1226331275 shutdown_connections 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 -- 172.21.15.153:0/1226331275 wait complete. 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient: build_initial_monmap 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient: monmap: 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:51.642 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 librados: starting msgr at 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 librados: starting objecter 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 Processor -- start 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 -- start start 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 librados: setting wanted keys 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 librados: calling monclient init 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient: init 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:34:51.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a140a41a0 0x7f8a14128aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient(hunting): picked mon.noname-a con 0x7f8a140a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a4580 0x7f8a14140920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient(hunting): picked mon.noname-c con 0x7f8a140a4580 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a140a49a0 0x7f8a141d0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient(hunting): picked mon.noname-b con 0x7f8a140a49a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f8a14144630 con 0x7f8a140a41a0 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f8a14070400 con 0x7f8a140a49a0 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f8a141447b0 con 0x7f8a140a4580 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a1a61e640 10 monclient(hunting): authenticate will time out at 1451.774572s 2024-09-10T20:34:51.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a140a41a0 0x7f8a14128aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a4580 0x7f8a14140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a140a41a0 0x7f8a14128aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56138/0 (socket says 172.21.15.153:56138) 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 1 -- 172.21.15.153:0/2590339471 learned_addr learned my addr 172.21.15.153:0/2590339471 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient(hunting): get_auth_request con 0x7f8a140a4580 auth_method 0 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 10 monclient(hunting): get_auth_request con 0x7f8a140a41a0 auth_method 0 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:51.645 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient(hunting): handle_auth_done global_id 54116 payload 824 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 1 -- 172.21.15.153:0/2590339471 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a140a49a0 msgr2=0x7f8a141d0ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 1 --2- 172.21.15.153:0/2590339471 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a140a49a0 0x7f8a141d0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 1 -- 172.21.15.153:0/2590339471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a140a41a0 msgr2=0x7f8a14128aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 1 --2- 172.21.15.153:0/2590339471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a140a41a0 0x7f8a14128aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 1 monclient: found mon.noname-c 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a0b7fe640 1 -- 172.21.15.153:0/2590339471 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a140a63a0 con 0x7f8a140a4580 2024-09-10T20:34:51.646 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.633+0000 7f8a13fff640 1 --2- 172.21.15.153:0/2590339471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a140a41a0 0x7f8a14128aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a0b7fe640 1 -- 172.21.15.153:0/2590339471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a4580 msgr2=0x7f8a14140920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a0b7fe640 1 --2- 172.21.15.153:0/2590339471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a4580 0x7f8a14140920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c3e0 tx=0x7f89fc00c8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 client.?.objecter ms_handle_connect 0x7f8a140a4580 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 1 -- 172.21.15.153:0/2590339471 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc0119a0 con 0x7f8a140a4580 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 monclient: dump: 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:51.647 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 monclient: _finish_auth 0 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 monclient: _check_auth_tickets 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 1 -- 172.21.15.153:0/2590339471 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89fc017070 con 0x7f8a140a4580 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 5 monclient: authenticate success, global_id 54116 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 10 monclient: _renew_subs 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a1412a2a0 con 0x7f8a140a4580 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:51.648 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 1 -- 172.21.15.153:0/2590339471 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89fc011d80 con 0x7f8a140a4580 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 monclient: dump: 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a12ffd640 4 set_mon_vals no callback set 2024-09-10T20:34:51.649 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 10 client.54116.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 10 monclient: _renew_subs 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a14129430 con 0x7f8a140a4580 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 1 librados: init done 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 1 -- 172.21.15.153:0/2590339471 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f89fc01c460 con 0x7f8a140a4580 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a140a3740 con 0x7f8a140a4580 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 1 --2- 172.21.15.153:0/2590339471 >> v2:172.21.15.176:6812/3003473398 conn(0x7f89f403cea0 0x7f89f403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 1 -- 172.21.15.153:0/2590339471 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f89fc04f380 con 0x7f8a140a4580 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 10 client.54116.objecter ms_dispatch 0x7f8a14000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 3 client.54116.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:51.650 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 3 client.54116.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:51.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a117fa640 20 client.54116.objecter dump_active .. 0 homeless 2024-09-10T20:34:51.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a13fff640 1 --2- 172.21.15.153:0/2590339471 >> v2:172.21.15.176:6812/3003473398 conn(0x7f89f403cea0 0x7f89f403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:51.651 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.637+0000 7f8a13fff640 10 monclient: get_auth_request con 0x7f89f403cea0 auth_method 0 2024-09-10T20:34:51.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.641+0000 7f8a13fff640 1 --2- 172.21.15.153:0/2590339471 >> v2:172.21.15.176:6812/3003473398 conn(0x7f89f403cea0 0x7f89f403f360 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009740 tx=0x7f8a04008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:51.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.641+0000 7f8a117fa640 10 client.54116.objecter ms_handle_connect 0x7f89f403cea0 2024-09-10T20:34:51.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.641+0000 7f8a117fa640 1 -- 172.21.15.153:0/2590339471 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f89fc04df20 con 0x7f8a140a4580 2024-09-10T20:34:51.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.641+0000 7f8a117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:51.652 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.641+0000 7f8a117fa640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:51.880 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.877+0000 7f8a1a61e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-10T20:34:51.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.877+0000 7f8a1a61e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-10T20:34:51.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.877+0000 7f8a1a61e640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:51.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.877+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8a1406cd40 con 0x7f8a140a4580 2024-09-10T20:34:51.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.877+0000 7f8a117fa640 1 -- 172.21.15.153:0/2590339471 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1568 (secure 0 0 0) 0x7f89fc04f020 con 0x7f8a140a4580 2024-09-10T20:34:51.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.877+0000 7f8a117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-10T20:34:51.881 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.877+0000 7f8a117fa640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:34:51.882 INFO:teuthology.orchestra.run.smithi153.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","modified":"2024-09-10T20:25:26.988641Z","created":"2024-09-10T20:25:26.988641Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3300","nonce":0},{"type":"v1","addr":"172.21.15.153:6789","nonce":0}]},"addr":"172.21.15.153:6789/0","public_addr":"172.21.15.153:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:3300","nonce":0},{"type":"v1","addr":"172.21.15.176:6789","nonce":0}]},"addr":"172.21.15.176:6789/0","public_addr":"172.21.15.176:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:3301","nonce":0},{"type":"v1","addr":"172.21.15.153:6790","nonce":0}]},"addr":"172.21.15.153:6790/0","public_addr":"172.21.15.153:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-10T20:34:51.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 10 librados: watch_flush enter 2024-09-10T20:34:51.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 10 librados: watch_flush exit 2024-09-10T20:34:51.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 20 client.54116.objecter shutdown clearing up homeless session... 2024-09-10T20:34:51.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 10 client.54116.objecter successfully canceled tick 2024-09-10T20:34:51.884 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 >> v2:172.21.15.176:6812/3003473398 conn(0x7f89f403cea0 msgr2=0x7f89f403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 --2- 172.21.15.153:0/2590339471 >> v2:172.21.15.176:6812/3003473398 conn(0x7f89f403cea0 0x7f89f403f360 secure :-1 s=READY pgs=3371 cs=0 l=1 rev1=1 crypto rx=0x7f8a04009740 tx=0x7f8a04008040 comp rx=0 tx=0).stop 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 10 monclient: shutdown 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a4580 msgr2=0x7f8a14140920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 --2- 172.21.15.153:0/2590339471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a4580 0x7f8a14140920 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f89fc00c3e0 tx=0x7f89fc00c8a0 comp rx=0 tx=0).stop 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 shutdown_connections 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 --2- 172.21.15.153:0/2590339471 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f8a140a49a0 0x7f8a141d0ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 --2- 172.21.15.153:0/2590339471 >> v2:172.21.15.176:6812/3003473398 conn(0x7f89f403cea0 0x7f89f403f360 unknown :-1 s=CLOSED pgs=3371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 --2- 172.21.15.153:0/2590339471 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f8a140a4580 0x7f8a14140920 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 --2- 172.21.15.153:0/2590339471 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f8a140a41a0 0x7f8a14128aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 >> 172.21.15.153:0/2590339471 conn(0x7f8a140ab7e0 msgr2=0x7f8a1409fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 shutdown_connections 2024-09-10T20:34:51.885 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 -- 172.21.15.153:0/2590339471 wait complete. 2024-09-10T20:34:51.886 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:51.881+0000 7f8a1a61e640 1 librados: shutdown 2024-09-10T20:34:51.900 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-10T20:34:51.900 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-10T20:34:51.909 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-10T20:34:51.909 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:34:52.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.021+0000 7fe848d2d640 10 monclient: get_monmap_and_config 2024-09-10T20:34:52.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.021+0000 7fe848d2d640 10 monclient: build_initial_monmap 2024-09-10T20:34:52.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.021+0000 7fe848d2d640 10 monclient: monmap: 2024-09-10T20:34:52.025 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:52.025 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.025 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:52.025056+0000 2024-09-10T20:34:52.025 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:52.025056+0000 2024-09-10T20:34:52.025 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:52.026 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.026 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:52.026 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:52.026 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:52.026 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:52.026 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:52.026 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 1 Processor -- start 2024-09-10T20:34:52.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 1 -- start start 2024-09-10T20:34:52.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 10 monclient: init 2024-09-10T20:34:52.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 10 monclient: _add_conns ranks=[4,1,0,3,2] 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe8440a41a0 0x7fe8440a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8440a41a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe8440a4580 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8440a49d0 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8440a67f0 con 0x7fe8440a49d0 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8440a6a20 con 0x7fe8440a4580 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe8440a2570 con 0x7fe8440a41a0 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe848d2d640 10 monclient(hunting): authenticate will time out at 1452.165176s 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe842d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe8440a49d0 0x7fe8440a60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60702/0 (socket says 172.21.15.153:60702) 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe842d9d640 1 -- 172.21.15.153:0/3292530308 learned_addr learned my addr 172.21.15.153:0/3292530308 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:52.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 1 --2- 172.21.15.153:0/3292530308 >> v2:172.21.15.176:3300/0 conn(0x7fe8440a41a0 0x7fe8440a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient(hunting): get_auth_request con 0x7fe8440a41a0 auth_method 0 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient(hunting): handle_auth_done global_id 20209 payload 274 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 1 -- 172.21.15.153:0/3292530308 >> v1:172.21.15.176:6789/0 conn(0x7fe8440a4580 legacy=0x7fe8440a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 1 -- 172.21.15.153:0/3292530308 >> v1:172.21.15.153:6789/0 conn(0x7fe8440a49d0 legacy=0x7fe8440a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient: _finish_hunting 0 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 1 monclient: found mon.noname-c 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:52.030 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:34:52.031 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.025+0000 7fe84259c640 1 -- 172.21.15.153:0/3292530308 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8440a3c80 con 0x7fe8440a41a0 2024-09-10T20:34:52.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84259c640 1 -- 172.21.15.153:0/3292530308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a41a0 msgr2=0x7fe8440a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:52.032 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84259c640 1 --2- 172.21.15.153:0/3292530308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a41a0 0x7fe8440a3740 secure :-1 s=READY pgs=5768 cs=0 l=1 rev1=1 crypto rx=0x7fe8340097c0 tx=0x7fe83401c360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0133cbdfdaabf6 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 1 -- 172.21.15.153:0/3292530308 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8340b0070 con 0x7fe8440a41a0 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 10 monclient: dump: 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.033 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.034 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.034 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.034 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 10 monclient: _finish_auth 0 2024-09-10T20:34:52.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 10 monclient: _check_auth_tickets 2024-09-10T20:34:52.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:52.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 1 -- 172.21.15.153:0/3292530308 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe8340ab600 con 0x7fe8440a41a0 2024-09-10T20:34:52.034 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 1 -- 172.21.15.153:0/3292530308 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8340ab9d0 con 0x7fe8440a41a0 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe84159a640 10 monclient: dump: 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.035 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe848d2d640 5 monclient: authenticate success, global_id 20209 2024-09-10T20:34:52.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe848d2d640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:52.036 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe848d2d640 4 set_mon_vals no callback set 2024-09-10T20:34:52.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe848d2d640 10 monclient: shutdown 2024-09-10T20:34:52.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe848d2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:52.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe848d2d640 1 -- 172.21.15.153:0/3292530308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a41a0 msgr2=0x7fe8440a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe848d2d640 1 --2- 172.21.15.153:0/3292530308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a41a0 0x7fe8440a3740 secure :-1 s=READY pgs=5768 cs=0 l=1 rev1=1 crypto rx=0x7fe8340097c0 tx=0x7fe83401c360 comp rx=0 tx=0).stop 2024-09-10T20:34:52.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe848d2d640 1 -- 172.21.15.153:0/3292530308 shutdown_connections 2024-09-10T20:34:52.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.029+0000 7fe848d2d640 1 --2- 172.21.15.153:0/3292530308 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a41a0 0x7fe8440a3740 unknown :-1 s=CLOSED pgs=5768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 -- 172.21.15.153:0/3292530308 >> 172.21.15.153:0/3292530308 conn(0x7fe8440ab7e0 msgr2=0x7fe8440a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:52.042 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 -- 172.21.15.153:0/3292530308 shutdown_connections 2024-09-10T20:34:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 -- 172.21.15.153:0/3292530308 wait complete. 2024-09-10T20:34:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient: build_initial_monmap 2024-09-10T20:34:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient: monmap: 2024-09-10T20:34:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:52.043 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 librados: starting msgr at 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 librados: starting objecter 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 Processor -- start 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 -- start start 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 librados: setting wanted keys 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 librados: calling monclient init 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient: init 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:52.044 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8440a41a0 0x7fe844128a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient(hunting): picked mon.noname-a con 0x7fe8440a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a4580 0x7fe844140880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient(hunting): picked mon.noname-b con 0x7fe8440a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8440a49d0 0x7fe8441d0990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient(hunting): picked mon.noname-c con 0x7fe8440a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe844144630 con 0x7fe8440a41a0 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe844070400 con 0x7fe8440a4580 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe8441447b0 con 0x7fe8440a49d0 2024-09-10T20:34:52.045 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe848d2d640 10 monclient(hunting): authenticate will time out at 1452.173867s 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8440a41a0 0x7fe844128a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8440a41a0 0x7fe844128a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56152/0 (socket says 172.21.15.153:56152) 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 1 -- 172.21.15.153:0/4035491475 learned_addr learned my addr 172.21.15.153:0/4035491475 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 10 monclient(hunting): get_auth_request con 0x7fe8440a41a0 auth_method 0 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8440a49d0 0x7fe8441d0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient(hunting): get_auth_request con 0x7fe8440a49d0 auth_method 0 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.046 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe841d9b640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a4580 0x7fe844140880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe841d9b640 10 monclient(hunting): get_auth_request con 0x7fe8440a4580 auth_method 0 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe841d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe841d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe841d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient(hunting): handle_auth_done global_id 54119 payload 824 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 1 -- 172.21.15.153:0/4035491475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a4580 msgr2=0x7fe844140880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.047 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a4580 0x7fe844140880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 1 -- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8440a41a0 msgr2=0x7fe844128a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8440a41a0 0x7fe844128a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient: _finish_hunting 0 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 1 monclient: found mon.noname-c 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe842d9d640 1 -- 172.21.15.153:0/4035491475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe844129f20 con 0x7fe8440a49d0 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.033+0000 7fe84259c640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8440a41a0 0x7fe844128a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe842d9d640 1 -- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8440a49d0 msgr2=0x7fe8441d0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe842d9d640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8440a49d0 0x7fe8441d0990 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00cac0 tx=0x7fe83c00cec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 client.?.objecter ms_handle_connect 0x7fe8440a49d0 2024-09-10T20:34:52.048 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 1 -- 172.21.15.153:0/4035491475 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c00ad80 con 0x7fe8440a49d0 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 monclient: dump: 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.049 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 monclient: _finish_auth 0 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 5 monclient: authenticate success, global_id 54119 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 10 monclient: _renew_subs 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8440a21c0 con 0x7fe8440a49d0 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 1 -- 172.21.15.153:0/4035491475 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe83c012400 con 0x7fe8440a49d0 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 1 -- 172.21.15.153:0/4035491475 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe83c0127f0 con 0x7fe8440a49d0 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.050 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 monclient: dump: 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe840d99640 4 set_mon_vals no callback set 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 10 client.54119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:52.051 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 10 monclient: _renew_subs 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe844127820 con 0x7fe8440a49d0 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 1 librados: init done 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8440a3740 con 0x7fe8440a49d0 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 1 -- 172.21.15.153:0/4035491475 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fe83c0129d0 con 0x7fe8440a49d0 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 1 --2- 172.21.15.153:0/4035491475 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe82403ced0 0x7fe82403f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 1 -- 172.21.15.153:0/4035491475 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7fe83c04fb00 con 0x7fe8440a49d0 2024-09-10T20:34:52.052 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 10 client.54119.objecter ms_dispatch 0x7fe844000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:52.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 3 client.54119.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:52.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 3 client.54119.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:52.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe832ffd640 20 client.54119.objecter dump_active .. 0 homeless 2024-09-10T20:34:52.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe84259c640 1 --2- 172.21.15.153:0/4035491475 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe82403ced0 0x7fe82403f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.053 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.037+0000 7fe84259c640 10 monclient: get_auth_request con 0x7fe82403ced0 auth_method 0 2024-09-10T20:34:52.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.041+0000 7fe84259c640 1 --2- 172.21.15.153:0/4035491475 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe82403ced0 0x7fe82403f390 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fe8340097c0 tx=0x7fe834008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:52.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.041+0000 7fe832ffd640 10 client.54119.objecter ms_handle_connect 0x7fe82403ced0 2024-09-10T20:34:52.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.041+0000 7fe832ffd640 1 -- 172.21.15.153:0/4035491475 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fe83c04e6a0 con 0x7fe8440a49d0 2024-09-10T20:34:52.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.041+0000 7fe832ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:52.054 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.041+0000 7fe832ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:52.204 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.201+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe84406cd40 con 0x7fe82403ced0 2024-09-10T20:34:52.210 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe832ffd640 1 -- 172.21.15.153:0/4035491475 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114145 (secure 0 0 0) 0x7fe84406cd40 con 0x7fe82403ced0 2024-09-10T20:34:52.212 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:34:52.212 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-10T20:34:52.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 10 librados: watch_flush enter 2024-09-10T20:34:52.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 10 librados: watch_flush exit 2024-09-10T20:34:52.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 20 client.54119.objecter shutdown clearing up homeless session... 2024-09-10T20:34:52.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 10 client.54119.objecter successfully canceled tick 2024-09-10T20:34:52.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe82403ced0 msgr2=0x7fe82403f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 --2- 172.21.15.153:0/4035491475 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe82403ced0 0x7fe82403f390 secure :-1 s=READY pgs=3372 cs=0 l=1 rev1=1 crypto rx=0x7fe8340097c0 tx=0x7fe834008040 comp rx=0 tx=0).stop 2024-09-10T20:34:52.214 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 10 monclient: shutdown 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8440a49d0 msgr2=0x7fe8441d0990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8440a49d0 0x7fe8441d0990 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe83c00cac0 tx=0x7fe83c00cec0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 shutdown_connections 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 --2- 172.21.15.153:0/4035491475 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe82403ced0 0x7fe82403f390 unknown :-1 s=CLOSED pgs=3372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8440a49d0 0x7fe8441d0990 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8440a4580 0x7fe844140880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 --2- 172.21.15.153:0/4035491475 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8440a41a0 0x7fe844128a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 >> 172.21.15.153:0/4035491475 conn(0x7fe8440ab7e0 msgr2=0x7fe8440a1540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 shutdown_connections 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 -- 172.21.15.153:0/4035491475 wait complete. 2024-09-10T20:34:52.215 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.209+0000 7fe848d2d640 1 librados: shutdown 2024-09-10T20:34:52.228 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":571,"stamp":"2024-09-10T20:34:51.249593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":449376,"kb_used_data":28984,"kb_used_omap":9,"kb_used_meta":420214,"kb_avail":565781664,"statfs":{"total":579820584960,"available":579360423936,"internally_reserved":0,"allocated":29679616,"data_stored":15420662,"data_compressed":596482,"data_compressed_allocated":966656,"data_compressed_original":3112960,"omap_allocated":9567,"internal_metadata":430299809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003835"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.828961+0000","last_change":"2024-09-10T20:27:17.198257+0000","last_active":"2024-09-10T20:33:29.828961+0000","last_peered":"2024-09-10T20:33:29.828961+0000","last_clean":"2024-09-10T20:33:29.828961+0000","last_became_active":"2024-09-10T20:27:17.197839+0000","last_became_peered":"2024-09-10T20:27:17.197839+0000","last_unstale":"2024-09-10T20:33:29.828961+0000","last_undegraded":"2024-09-10T20:33:29.828961+0000","last_fullsized":"2024-09-10T20:33:29.828961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:15.959864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500272+0000","last_change":"2024-09-10T20:27:17.223318+0000","last_active":"2024-09-10T20:33:29.500272+0000","last_peered":"2024-09-10T20:33:29.500272+0000","last_clean":"2024-09-10T20:33:29.500272+0000","last_became_active":"2024-09-10T20:27:17.219498+0000","last_became_peered":"2024-09-10T20:27:17.219498+0000","last_unstale":"2024-09-10T20:33:29.500272+0000","last_undegraded":"2024-09-10T20:33:29.500272+0000","last_fullsized":"2024-09-10T20:33:29.500272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:12.081089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498568+0000","last_change":"2024-09-10T20:27:17.202919+0000","last_active":"2024-09-10T20:33:29.498568+0000","last_peered":"2024-09-10T20:33:29.498568+0000","last_clean":"2024-09-10T20:33:29.498568+0000","last_became_active":"2024-09-10T20:27:17.202373+0000","last_became_peered":"2024-09-10T20:27:17.202373+0000","last_unstale":"2024-09-10T20:33:29.498568+0000","last_undegraded":"2024-09-10T20:33:29.498568+0000","last_fullsized":"2024-09-10T20:33:29.498568+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:42:27.202536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500829+0000","last_change":"2024-09-10T20:27:17.218042+0000","last_active":"2024-09-10T20:33:29.500829+0000","last_peered":"2024-09-10T20:33:29.500829+0000","last_clean":"2024-09-10T20:33:29.500829+0000","last_became_active":"2024-09-10T20:27:17.217517+0000","last_became_peered":"2024-09-10T20:27:17.217517+0000","last_unstale":"2024-09-10T20:33:29.500829+0000","last_undegraded":"2024-09-10T20:33:29.500829+0000","last_fullsized":"2024-09-10T20:33:29.500829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:53.695011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829236+0000","last_change":"2024-09-10T20:27:17.282375+0000","last_active":"2024-09-10T20:33:29.829236+0000","last_peered":"2024-09-10T20:33:29.829236+0000","last_clean":"2024-09-10T20:33:29.829236+0000","last_became_active":"2024-09-10T20:27:17.281836+0000","last_became_peered":"2024-09-10T20:27:17.281836+0000","last_unstale":"2024-09-10T20:33:29.829236+0000","last_undegraded":"2024-09-10T20:33:29.829236+0000","last_fullsized":"2024-09-10T20:33:29.829236+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:51:27.168991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498729+0000","last_change":"2024-09-10T20:27:17.213190+0000","last_active":"2024-09-10T20:33:29.498729+0000","last_peered":"2024-09-10T20:33:29.498729+0000","last_clean":"2024-09-10T20:33:29.498729+0000","last_became_active":"2024-09-10T20:27:17.212620+0000","last_became_peered":"2024-09-10T20:27:17.212620+0000","last_unstale":"2024-09-10T20:33:29.498729+0000","last_undegraded":"2024-09-10T20:33:29.498729+0000","last_fullsized":"2024-09-10T20:33:29.498729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:48.977530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498751+0000","last_change":"2024-09-10T20:27:17.288663+0000","last_active":"2024-09-10T20:33:29.498751+0000","last_peered":"2024-09-10T20:33:29.498751+0000","last_clean":"2024-09-10T20:33:29.498751+0000","last_became_active":"2024-09-10T20:27:17.287973+0000","last_became_peered":"2024-09-10T20:27:17.287973+0000","last_unstale":"2024-09-10T20:33:29.498751+0000","last_undegraded":"2024-09-10T20:33:29.498751+0000","last_fullsized":"2024-09-10T20:33:29.498751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:16:25.413353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874217+0000","last_change":"2024-09-10T20:27:17.210079+0000","last_active":"2024-09-10T20:33:29.874217+0000","last_peered":"2024-09-10T20:33:29.874217+0000","last_clean":"2024-09-10T20:33:29.874217+0000","last_became_active":"2024-09-10T20:27:17.209693+0000","last_became_peered":"2024-09-10T20:27:17.209693+0000","last_unstale":"2024-09-10T20:33:29.874217+0000","last_undegraded":"2024-09-10T20:33:29.874217+0000","last_fullsized":"2024-09-10T20:33:29.874217+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:44.983674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500800+0000","last_change":"2024-09-10T20:27:17.249972+0000","last_active":"2024-09-10T20:33:29.500800+0000","last_peered":"2024-09-10T20:33:29.500800+0000","last_clean":"2024-09-10T20:33:29.500800+0000","last_became_active":"2024-09-10T20:27:17.249683+0000","last_became_peered":"2024-09-10T20:27:17.249683+0000","last_unstale":"2024-09-10T20:33:29.500800+0000","last_undegraded":"2024-09-10T20:33:29.500800+0000","last_fullsized":"2024-09-10T20:33:29.500800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:02:11.684811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498432+0000","last_change":"2024-09-10T20:27:17.248264+0000","last_active":"2024-09-10T20:33:29.498432+0000","last_peered":"2024-09-10T20:33:29.498432+0000","last_clean":"2024-09-10T20:33:29.498432+0000","last_became_active":"2024-09-10T20:27:17.247865+0000","last_became_peered":"2024-09-10T20:27:17.247865+0000","last_unstale":"2024-09-10T20:33:29.498432+0000","last_undegraded":"2024-09-10T20:33:29.498432+0000","last_fullsized":"2024-09-10T20:33:29.498432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:06:33.643650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829007+0000","last_change":"2024-09-10T20:27:17.203381+0000","last_active":"2024-09-10T20:33:29.829007+0000","last_peered":"2024-09-10T20:33:29.829007+0000","last_clean":"2024-09-10T20:33:29.829007+0000","last_became_active":"2024-09-10T20:27:17.203002+0000","last_became_peered":"2024-09-10T20:27:17.203002+0000","last_unstale":"2024-09-10T20:33:29.829007+0000","last_undegraded":"2024-09-10T20:33:29.829007+0000","last_fullsized":"2024-09-10T20:33:29.829007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:32:20.452158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873964+0000","last_change":"2024-09-10T20:27:17.298627+0000","last_active":"2024-09-10T20:33:29.873964+0000","last_peered":"2024-09-10T20:33:29.873964+0000","last_clean":"2024-09-10T20:33:29.873964+0000","last_became_active":"2024-09-10T20:27:17.298309+0000","last_became_peered":"2024-09-10T20:27:17.298309+0000","last_unstale":"2024-09-10T20:33:29.873964+0000","last_undegraded":"2024-09-10T20:33:29.873964+0000","last_fullsized":"2024-09-10T20:33:29.873964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:55.361013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":596,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500685+0000","last_change":"2024-09-10T20:26:21.773251+0000","last_active":"2024-09-10T20:33:29.500685+0000","last_peered":"2024-09-10T20:33:29.500685+0000","last_clean":"2024-09-10T20:33:29.500685+0000","last_became_active":"2024-09-10T20:26:19.356844+0000","last_became_peered":"2024-09-10T20:26:19.356844+0000","last_unstale":"2024-09-10T20:33:29.500685+0000","last_undegraded":"2024-09-10T20:33:29.500685+0000","last_fullsized":"2024-09-10T20:33:29.500685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:23.413224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873697+0000","last_change":"2024-09-10T20:27:17.206032+0000","last_active":"2024-09-10T20:33:29.873697+0000","last_peered":"2024-09-10T20:33:29.873697+0000","last_clean":"2024-09-10T20:33:29.873697+0000","last_became_active":"2024-09-10T20:27:17.205464+0000","last_became_peered":"2024-09-10T20:27:17.205464+0000","last_unstale":"2024-09-10T20:33:29.873697+0000","last_undegraded":"2024-09-10T20:33:29.873697+0000","last_fullsized":"2024-09-10T20:33:29.873697+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:33.219827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":593,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829238+0000","last_change":"2024-09-10T20:26:21.381255+0000","last_active":"2024-09-10T20:33:29.829238+0000","last_peered":"2024-09-10T20:33:29.829238+0000","last_clean":"2024-09-10T20:33:29.829238+0000","last_became_active":"2024-09-10T20:26:19.363220+0000","last_became_peered":"2024-09-10T20:26:19.363220+0000","last_unstale":"2024-09-10T20:33:29.829238+0000","last_undegraded":"2024-09-10T20:33:29.829238+0000","last_fullsized":"2024-09-10T20:33:29.829238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:09.279291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096679600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"23'7","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500753+0000","last_change":"2024-09-10T20:27:17.217628+0000","last_active":"2024-09-10T20:33:29.500753+0000","last_peered":"2024-09-10T20:33:29.500753+0000","last_clean":"2024-09-10T20:33:29.500753+0000","last_became_active":"2024-09-10T20:27:17.211037+0000","last_became_peered":"2024-09-10T20:27:17.211037+0000","last_unstale":"2024-09-10T20:33:29.500753+0000","last_undegraded":"2024-09-10T20:33:29.500753+0000","last_fullsized":"2024-09-10T20:33:29.500753+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:47.151425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":597,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500610+0000","last_change":"2024-09-10T20:26:21.773020+0000","last_active":"2024-09-10T20:33:29.500610+0000","last_peered":"2024-09-10T20:33:29.500610+0000","last_clean":"2024-09-10T20:33:29.500610+0000","last_became_active":"2024-09-10T20:26:19.363294+0000","last_became_peered":"2024-09-10T20:26:19.363294+0000","last_unstale":"2024-09-10T20:33:29.500610+0000","last_undegraded":"2024-09-10T20:33:29.500610+0000","last_fullsized":"2024-09-10T20:33:29.500610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:54.300619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053952999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.875890+0000","last_change":"2024-09-10T20:27:17.298070+0000","last_active":"2024-09-10T20:33:29.875890+0000","last_peered":"2024-09-10T20:33:29.875890+0000","last_clean":"2024-09-10T20:33:29.875890+0000","last_became_active":"2024-09-10T20:27:17.297745+0000","last_became_peered":"2024-09-10T20:27:17.297745+0000","last_unstale":"2024-09-10T20:33:29.875890+0000","last_undegraded":"2024-09-10T20:33:29.875890+0000","last_fullsized":"2024-09-10T20:33:29.875890+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:30:22.598793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874003+0000","last_change":"2024-09-10T20:26:21.378377+0000","last_active":"2024-09-10T20:33:29.874003+0000","last_peered":"2024-09-10T20:33:29.874003+0000","last_clean":"2024-09-10T20:33:29.874003+0000","last_became_active":"2024-09-10T20:26:19.359961+0000","last_became_peered":"2024-09-10T20:26:19.359961+0000","last_unstale":"2024-09-10T20:33:29.874003+0000","last_undegraded":"2024-09-10T20:33:29.874003+0000","last_fullsized":"2024-09-10T20:33:29.874003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:21:01.940376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829065+0000","last_change":"2024-09-10T20:27:17.282394+0000","last_active":"2024-09-10T20:33:29.829065+0000","last_peered":"2024-09-10T20:33:29.829065+0000","last_clean":"2024-09-10T20:33:29.829065+0000","last_became_active":"2024-09-10T20:27:17.281815+0000","last_became_peered":"2024-09-10T20:27:17.281815+0000","last_unstale":"2024-09-10T20:33:29.829065+0000","last_undegraded":"2024-09-10T20:33:29.829065+0000","last_fullsized":"2024-09-10T20:33:29.829065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:49.830688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":597,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876148+0000","last_change":"2024-09-10T20:26:21.386697+0000","last_active":"2024-09-10T20:33:29.876148+0000","last_peered":"2024-09-10T20:33:29.876148+0000","last_clean":"2024-09-10T20:33:29.876148+0000","last_became_active":"2024-09-10T20:26:19.360427+0000","last_became_peered":"2024-09-10T20:26:19.360427+0000","last_unstale":"2024-09-10T20:33:29.876148+0000","last_undegraded":"2024-09-10T20:33:29.876148+0000","last_fullsized":"2024-09-10T20:33:29.876148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:53.136006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020581079999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873800+0000","last_change":"2024-09-10T20:27:17.282845+0000","last_active":"2024-09-10T20:33:29.873800+0000","last_peered":"2024-09-10T20:33:29.873800+0000","last_clean":"2024-09-10T20:33:29.873800+0000","last_became_active":"2024-09-10T20:27:17.282600+0000","last_became_peered":"2024-09-10T20:27:17.282600+0000","last_unstale":"2024-09-10T20:33:29.873800+0000","last_undegraded":"2024-09-10T20:33:29.873800+0000","last_fullsized":"2024-09-10T20:33:29.873800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:38.719966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":603,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873765+0000","last_change":"2024-09-10T20:26:21.575187+0000","last_active":"2024-09-10T20:33:29.873765+0000","last_peered":"2024-09-10T20:33:29.873765+0000","last_clean":"2024-09-10T20:33:29.873765+0000","last_became_active":"2024-09-10T20:26:19.362858+0000","last_became_peered":"2024-09-10T20:26:19.362858+0000","last_unstale":"2024-09-10T20:33:29.873765+0000","last_undegraded":"2024-09-10T20:33:29.873765+0000","last_fullsized":"2024-09-10T20:33:29.873765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:22:17.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078714099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829346+0000","last_change":"2024-09-10T20:27:17.214784+0000","last_active":"2024-09-10T20:33:29.829346+0000","last_peered":"2024-09-10T20:33:29.829346+0000","last_clean":"2024-09-10T20:33:29.829346+0000","last_became_active":"2024-09-10T20:27:17.214428+0000","last_became_peered":"2024-09-10T20:27:17.214428+0000","last_unstale":"2024-09-10T20:33:29.829346+0000","last_undegraded":"2024-09-10T20:33:29.829346+0000","last_fullsized":"2024-09-10T20:33:29.829346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:50:15.983575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":610,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498452+0000","last_change":"2024-09-10T20:26:21.378915+0000","last_active":"2024-09-10T20:33:29.498452+0000","last_peered":"2024-09-10T20:33:29.498452+0000","last_clean":"2024-09-10T20:33:29.498452+0000","last_became_active":"2024-09-10T20:26:19.362158+0000","last_became_peered":"2024-09-10T20:26:19.362158+0000","last_unstale":"2024-09-10T20:33:29.498452+0000","last_undegraded":"2024-09-10T20:33:29.498452+0000","last_fullsized":"2024-09-10T20:33:29.498452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:50.708554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010122729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876226+0000","last_change":"2024-09-10T20:27:17.244860+0000","last_active":"2024-09-10T20:33:29.876226+0000","last_peered":"2024-09-10T20:33:29.876226+0000","last_clean":"2024-09-10T20:33:29.876226+0000","last_became_active":"2024-09-10T20:27:17.243935+0000","last_became_peered":"2024-09-10T20:27:17.243935+0000","last_unstale":"2024-09-10T20:33:29.876226+0000","last_undegraded":"2024-09-10T20:33:29.876226+0000","last_fullsized":"2024-09-10T20:33:29.876226+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:11.085907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":598,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500566+0000","last_change":"2024-09-10T20:26:21.773025+0000","last_active":"2024-09-10T20:33:29.500566+0000","last_peered":"2024-09-10T20:33:29.500566+0000","last_clean":"2024-09-10T20:33:29.500566+0000","last_became_active":"2024-09-10T20:26:19.359775+0000","last_became_peered":"2024-09-10T20:26:19.359775+0000","last_unstale":"2024-09-10T20:33:29.500566+0000","last_undegraded":"2024-09-10T20:33:29.500566+0000","last_fullsized":"2024-09-10T20:33:29.500566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:35.427907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054594699999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":688,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500496+0000","last_change":"2024-09-10T20:26:14.579551+0000","last_active":"2024-09-10T20:33:29.500496+0000","last_peered":"2024-09-10T20:33:29.500496+0000","last_clean":"2024-09-10T20:33:29.500496+0000","last_became_active":"2024-09-10T20:26:14.577602+0000","last_became_peered":"2024-09-10T20:26:14.577602+0000","last_unstale":"2024-09-10T20:33:29.500496+0000","last_undegraded":"2024-09-10T20:33:29.500496+0000","last_fullsized":"2024-09-10T20:33:29.500496+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_clean_scrub_stamp":"2024-09-10T20:26:13.287430+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:41.463707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.875916+0000","last_change":"2024-09-10T20:27:17.297570+0000","last_active":"2024-09-10T20:33:29.875916+0000","last_peered":"2024-09-10T20:33:29.875916+0000","last_clean":"2024-09-10T20:33:29.875916+0000","last_became_active":"2024-09-10T20:27:17.295709+0000","last_became_peered":"2024-09-10T20:27:17.295709+0000","last_unstale":"2024-09-10T20:33:29.875916+0000","last_undegraded":"2024-09-10T20:33:29.875916+0000","last_fullsized":"2024-09-10T20:33:29.875916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:37.519789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498888+0000","last_change":"2024-09-10T20:27:17.219710+0000","last_active":"2024-09-10T20:33:29.498888+0000","last_peered":"2024-09-10T20:33:29.498888+0000","last_clean":"2024-09-10T20:33:29.498888+0000","last_became_active":"2024-09-10T20:27:17.217973+0000","last_became_peered":"2024-09-10T20:27:17.217973+0000","last_unstale":"2024-09-10T20:33:29.498888+0000","last_undegraded":"2024-09-10T20:33:29.498888+0000","last_fullsized":"2024-09-10T20:33:29.498888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:55.139486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500679+0000","last_change":"2024-09-10T20:27:17.318458+0000","last_active":"2024-09-10T20:33:29.500679+0000","last_peered":"2024-09-10T20:33:29.500679+0000","last_clean":"2024-09-10T20:33:29.500679+0000","last_became_active":"2024-09-10T20:27:17.318202+0000","last_became_peered":"2024-09-10T20:27:17.318202+0000","last_unstale":"2024-09-10T20:33:29.500679+0000","last_undegraded":"2024-09-10T20:33:29.500679+0000","last_fullsized":"2024-09-10T20:33:29.500679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:49.111756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873865+0000","last_change":"2024-09-10T20:27:17.240732+0000","last_active":"2024-09-10T20:33:29.873865+0000","last_peered":"2024-09-10T20:33:29.873865+0000","last_clean":"2024-09-10T20:33:29.873865+0000","last_became_active":"2024-09-10T20:27:17.240435+0000","last_became_peered":"2024-09-10T20:27:17.240435+0000","last_unstale":"2024-09-10T20:33:29.873865+0000","last_undegraded":"2024-09-10T20:33:29.873865+0000","last_fullsized":"2024-09-10T20:33:29.873865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:54:42.900319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874276+0000","last_change":"2024-09-10T20:27:17.230157+0000","last_active":"2024-09-10T20:33:29.874276+0000","last_peered":"2024-09-10T20:33:29.874276+0000","last_clean":"2024-09-10T20:33:29.874276+0000","last_became_active":"2024-09-10T20:27:17.229886+0000","last_became_peered":"2024-09-10T20:27:17.229886+0000","last_unstale":"2024-09-10T20:33:29.874276+0000","last_undegraded":"2024-09-10T20:33:29.874276+0000","last_fullsized":"2024-09-10T20:33:29.874276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:40.265668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874302+0000","last_change":"2024-09-10T20:27:17.214355+0000","last_active":"2024-09-10T20:33:29.874302+0000","last_peered":"2024-09-10T20:33:29.874302+0000","last_clean":"2024-09-10T20:33:29.874302+0000","last_became_active":"2024-09-10T20:27:17.213875+0000","last_became_peered":"2024-09-10T20:27:17.213875+0000","last_unstale":"2024-09-10T20:33:29.874302+0000","last_undegraded":"2024-09-10T20:33:29.874302+0000","last_fullsized":"2024-09-10T20:33:29.874302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:55.411553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873798+0000","last_change":"2024-09-10T20:27:17.288406+0000","last_active":"2024-09-10T20:33:29.873798+0000","last_peered":"2024-09-10T20:33:29.873798+0000","last_clean":"2024-09-10T20:33:29.873798+0000","last_became_active":"2024-09-10T20:27:17.288092+0000","last_became_peered":"2024-09-10T20:27:17.288092+0000","last_unstale":"2024-09-10T20:33:29.873798+0000","last_undegraded":"2024-09-10T20:33:29.873798+0000","last_fullsized":"2024-09-10T20:33:29.873798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:47.584995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873374+0000","last_change":"2024-09-10T20:27:17.210249+0000","last_active":"2024-09-10T20:33:29.873374+0000","last_peered":"2024-09-10T20:33:29.873374+0000","last_clean":"2024-09-10T20:33:29.873374+0000","last_became_active":"2024-09-10T20:27:17.209363+0000","last_became_peered":"2024-09-10T20:27:17.209363+0000","last_unstale":"2024-09-10T20:33:29.873374+0000","last_undegraded":"2024-09-10T20:33:29.873374+0000","last_fullsized":"2024-09-10T20:33:29.873374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:19.942713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500468+0000","last_change":"2024-09-10T20:27:17.225434+0000","last_active":"2024-09-10T20:33:29.500468+0000","last_peered":"2024-09-10T20:33:29.500468+0000","last_clean":"2024-09-10T20:33:29.500468+0000","last_became_active":"2024-09-10T20:27:17.224230+0000","last_became_peered":"2024-09-10T20:27:17.224230+0000","last_unstale":"2024-09-10T20:33:29.500468+0000","last_undegraded":"2024-09-10T20:33:29.500468+0000","last_fullsized":"2024-09-10T20:33:29.500468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:49:09.800954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500670+0000","last_change":"2024-09-10T20:27:17.217944+0000","last_active":"2024-09-10T20:33:29.500670+0000","last_peered":"2024-09-10T20:33:29.500670+0000","last_clean":"2024-09-10T20:33:29.500670+0000","last_became_active":"2024-09-10T20:27:17.217107+0000","last_became_peered":"2024-09-10T20:27:17.217107+0000","last_unstale":"2024-09-10T20:33:29.500670+0000","last_undegraded":"2024-09-10T20:33:29.500670+0000","last_fullsized":"2024-09-10T20:33:29.500670+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:52.708045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829272+0000","last_change":"2024-09-10T20:27:17.283023+0000","last_active":"2024-09-10T20:33:29.829272+0000","last_peered":"2024-09-10T20:33:29.829272+0000","last_clean":"2024-09-10T20:33:29.829272+0000","last_became_active":"2024-09-10T20:27:17.282074+0000","last_became_peered":"2024-09-10T20:27:17.282074+0000","last_unstale":"2024-09-10T20:33:29.829272+0000","last_undegraded":"2024-09-10T20:33:29.829272+0000","last_fullsized":"2024-09-10T20:33:29.829272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:51.986119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874322+0000","last_change":"2024-09-10T20:27:17.211220+0000","last_active":"2024-09-10T20:33:29.874322+0000","last_peered":"2024-09-10T20:33:29.874322+0000","last_clean":"2024-09-10T20:33:29.874322+0000","last_became_active":"2024-09-10T20:27:17.210803+0000","last_became_peered":"2024-09-10T20:27:17.210803+0000","last_unstale":"2024-09-10T20:33:29.874322+0000","last_undegraded":"2024-09-10T20:33:29.874322+0000","last_fullsized":"2024-09-10T20:33:29.874322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:38.315319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876046+0000","last_change":"2024-09-10T20:27:17.311360+0000","last_active":"2024-09-10T20:33:29.876046+0000","last_peered":"2024-09-10T20:33:29.876046+0000","last_clean":"2024-09-10T20:33:29.876046+0000","last_became_active":"2024-09-10T20:27:17.310182+0000","last_became_peered":"2024-09-10T20:27:17.310182+0000","last_unstale":"2024-09-10T20:33:29.876046+0000","last_undegraded":"2024-09-10T20:33:29.876046+0000","last_fullsized":"2024-09-10T20:33:29.876046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:34:02.949919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574956,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80416,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291424,"statfs":{"total":96636764160,"available":96554418176,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80488,"kb_used_data":4872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291352,"statfs":{"total":96636764160,"available":96554344448,"internally_reserved":0,"allocated":4988928,"data_stored":2922726,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574953,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76260,"kb_used_data":4812,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295580,"statfs":{"total":96636764160,"available":96558673920,"internally_reserved":0,"allocated":4927488,"data_stored":2393812,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640363,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72172,"kb_used_data":4868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299668,"statfs":{"total":96636764160,"available":96562860032,"internally_reserved":0,"allocated":4984832,"data_stored":2922721,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67944,"kb_used_data":4816,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303896,"statfs":{"total":96636764160,"available":96567189504,"internally_reserved":0,"allocated":4931584,"data_stored":2393817,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607658,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72096,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299744,"statfs":{"total":96636764160,"available":96562937856,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:34:52.230 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:34:52.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.341+0000 7f3d29217640 10 monclient: get_monmap_and_config 2024-09-10T20:34:52.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.341+0000 7f3d29217640 10 monclient: build_initial_monmap 2024-09-10T20:34:52.345 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.341+0000 7f3d29217640 10 monclient: monmap: 2024-09-10T20:34:52.345 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:52.345 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.345 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:52.345163+0000 2024-09-10T20:34:52.346 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:52.345163+0000 2024-09-10T20:34:52.346 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:52.346 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.346 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:52.346 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:52.346 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:52.346 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:52.346 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:52.346 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 1 Processor -- start 2024-09-10T20:34:52.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 1 -- start start 2024-09-10T20:34:52.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient: init 2024-09-10T20:34:52.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:52.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient: _add_conns ranks=[1,4,2,0,3] 2024-09-10T20:34:52.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3d240a41a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f3d240a4580 0x7f3d2413d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient(hunting): picked mon.noname-c con 0x7f3d240a4580 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient(hunting): picked mon.noname-b con 0x7f3d240a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d240a6a20 con 0x7f3d240a41a0 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f3d240a2570 con 0x7f3d240a4580 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3d24070400 con 0x7f3d240a49d0 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d29217640 10 monclient(hunting): authenticate will time out at 1452.484978s 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37356/0 (socket says 172.21.15.153:37356) 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 1 -- 172.21.15.153:0/1627631594 learned_addr learned my addr 172.21.15.153:0/1627631594 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient(hunting): get_auth_request con 0x7f3d240a49d0 auth_method 0 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d1bfff640 1 --2- 172.21.15.153:0/1627631594 >> v2:172.21.15.176:3300/0 conn(0x7f3d240a4580 0x7f3d2413d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d1bfff640 10 monclient(hunting): get_auth_request con 0x7f3d240a4580 auth_method 0 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient(hunting): handle_auth_done global_id 54122 payload 274 2024-09-10T20:34:52.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 1 -- 172.21.15.153:0/1627631594 >> v2:172.21.15.176:3300/0 conn(0x7f3d240a4580 msgr2=0x7f3d2413d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:52.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 1 --2- 172.21.15.153:0/1627631594 >> v2:172.21.15.176:3300/0 conn(0x7f3d240a4580 0x7f3d2413d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 1 -- 172.21.15.153:0/1627631594 >> v1:172.21.15.176:6789/0 conn(0x7f3d240a41a0 legacy=0x7f3d240a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:52.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient: _finish_hunting 0 2024-09-10T20:34:52.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 1 monclient: found mon.noname-b 2024-09-10T20:34:52.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:52.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:52.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.345+0000 7f3d2359e640 1 -- 172.21.15.153:0/1627631594 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d240a85a0 con 0x7f3d240a49d0 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2359e640 1 -- 172.21.15.153:0/1627631594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 msgr2=0x7f3d2413fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2359e640 1 --2- 172.21.15.153:0/1627631594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413fd10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009e50 tx=0x7f3d1c01c8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d57a81d0e5dae4e1 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 1 -- 172.21.15.153:0/1627631594 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0a5730 con 0x7f3d240a49d0 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 10 monclient: dump: 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.355 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 10 monclient: _finish_auth 0 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 10 monclient: _check_auth_tickets 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 1 -- 172.21.15.153:0/1627631594 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0a58d0 con 0x7f3d240a49d0 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 1 -- 172.21.15.153:0/1627631594 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0a5ca0 con 0x7f3d240a49d0 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2259c640 10 monclient: dump: 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.356 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 5 monclient: authenticate success, global_id 54122 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 4 set_mon_vals no callback set 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient: shutdown 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:52.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 -- 172.21.15.153:0/1627631594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 msgr2=0x7f3d2413fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 --2- 172.21.15.153:0/1627631594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413fd10 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009e50 tx=0x7f3d1c01c8f0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 -- 172.21.15.153:0/1627631594 shutdown_connections 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 --2- 172.21.15.153:0/1627631594 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413fd10 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 --2- 172.21.15.153:0/1627631594 >> v2:172.21.15.176:3300/0 conn(0x7f3d240a4580 0x7f3d2413d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 -- 172.21.15.153:0/1627631594 >> 172.21.15.153:0/1627631594 conn(0x7f3d240ab7e0 msgr2=0x7f3d240a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 -- 172.21.15.153:0/1627631594 shutdown_connections 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 -- 172.21.15.153:0/1627631594 wait complete. 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient: build_initial_monmap 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient: monmap: 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:52.358 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 librados: starting msgr at 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 librados: starting objecter 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 Processor -- start 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 -- start start 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 librados: setting wanted keys 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 librados: calling monclient init 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient: init 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:52.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d240a41a0 0x7f3d24128460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient(hunting): picked mon.noname-b con 0x7f3d240a41a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240a4580 0x7f3d24126300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient(hunting): picked mon.noname-a con 0x7f3d240a4580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient(hunting): picked mon.noname-c con 0x7f3d240a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f3d24142e00 con 0x7f3d240a4580 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f3d24142c80 con 0x7f3d240a41a0 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f3d24142f80 con 0x7f3d240a49d0 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:52.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d29217640 10 monclient(hunting): authenticate will time out at 1452.491835s 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2359e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d1bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240a4580 0x7f3d24126300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2359e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37362/0 (socket says 172.21.15.153:37362) 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d1bfff640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240a4580 0x7f3d24126300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56166/0 (socket says 172.21.15.153:56166) 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2359e640 1 -- 172.21.15.153:0/3720235277 learned_addr learned my addr 172.21.15.153:0/3720235277 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2359e640 10 monclient(hunting): get_auth_request con 0x7f3d240a49d0 auth_method 0 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2359e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2359e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.349+0000 7f3d2359e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1bfff640 10 monclient(hunting): get_auth_request con 0x7f3d240a4580 auth_method 0 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1bfff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1bfff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1bfff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1bfff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1bfff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d22d9d640 1 --2- 172.21.15.153:0/3720235277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d240a41a0 0x7f3d24128460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d22d9d640 10 monclient(hunting): get_auth_request con 0x7f3d240a41a0 auth_method 0 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d22d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d22d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 10 monclient(hunting): handle_auth_done global_id 54125 payload 824 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 1 -- 172.21.15.153:0/3720235277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d240a41a0 msgr2=0x7f3d24128460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 1 --2- 172.21.15.153:0/3720235277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d240a41a0 0x7f3d24128460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 1 -- 172.21.15.153:0/3720235277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240a4580 msgr2=0x7f3d24126300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 1 --2- 172.21.15.153:0/3720235277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240a4580 0x7f3d24126300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 10 monclient: _finish_hunting 0 2024-09-10T20:34:52.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 1 monclient: found mon.noname-c 2024-09-10T20:34:52.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:52.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:52.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 1 -- 172.21.15.153:0/3720235277 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d240a21c0 con 0x7f3d240a49d0 2024-09-10T20:34:52.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1bfff640 1 --2- 172.21.15.153:0/3720235277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240a4580 0x7f3d24126300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:52.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 1 -- 172.21.15.153:0/3720235277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 msgr2=0x7f3d2413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:52.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d2359e640 1 --2- 172.21.15.153:0/3720235277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413f2d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0af040 tx=0x7f3d1c019690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:52.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 client.?.objecter ms_handle_connect 0x7f3d240a49d0 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 1 -- 172.21.15.153:0/3720235277 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0bc020 con 0x7f3d240a49d0 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 monclient: dump: 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.366 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 monclient: _finish_auth 0 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 monclient: _check_auth_tickets 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 5 monclient: authenticate success, global_id 54125 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 1 -- 172.21.15.153:0/3720235277 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1c0b8a90 con 0x7f3d240a49d0 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 10 monclient: _renew_subs 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 1 -- 172.21.15.153:0/3720235277 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c0b8e60 con 0x7f3d240a49d0 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d24127900 con 0x7f3d240a49d0 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d21d9b640 4 set_mon_vals no callback set 2024-09-10T20:34:52.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d1b7fe640 10 monclient: dump: 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 10 client.54125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:52.368 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 10 monclient: _renew_subs 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d241296a0 con 0x7f3d240a49d0 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 1 librados: init done 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.353+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d24140030 con 0x7f3d240a49d0 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d1b7fe640 1 -- 172.21.15.153:0/3720235277 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f3d1c0bf020 con 0x7f3d240a49d0 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d1b7fe640 1 --2- 172.21.15.153:0/3720235277 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3d0403cef0 0x7f3d0403f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d1b7fe640 1 -- 172.21.15.153:0/3720235277 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f3d1c0eaad0 con 0x7f3d240a49d0 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d1b7fe640 10 client.54125.objecter ms_dispatch 0x7f3d24000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d1b7fe640 3 client.54125.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d1b7fe640 3 client.54125.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:52.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d1b7fe640 20 client.54125.objecter dump_active .. 0 homeless 2024-09-10T20:34:52.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d22d9d640 1 --2- 172.21.15.153:0/3720235277 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3d0403cef0 0x7f3d0403f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d22d9d640 10 monclient: get_auth_request con 0x7f3d0403cef0 auth_method 0 2024-09-10T20:34:52.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d22d9d640 1 --2- 172.21.15.153:0/3720235277 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3d0403cef0 0x7f3d0403f3b0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f3d14004e30 tx=0x7f3d14009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:52.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.357+0000 7f3d1b7fe640 10 client.54125.objecter ms_handle_connect 0x7f3d0403cef0 2024-09-10T20:34:52.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.361+0000 7f3d1b7fe640 1 -- 172.21.15.153:0/3720235277 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f3d1c0c6030 con 0x7f3d240a49d0 2024-09-10T20:34:52.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.361+0000 7f3d1b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:52.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.361+0000 7f3d1b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:52.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.525+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3d241298e0 con 0x7f3d0403cef0 2024-09-10T20:34:52.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.529+0000 7f3d1b7fe640 1 -- 172.21.15.153:0/3720235277 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114145 (secure 0 0 0) 0x7f3d140053b0 con 0x7f3d0403cef0 2024-09-10T20:34:52.534 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:34:52.535 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-10T20:34:52.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 10 librados: watch_flush enter 2024-09-10T20:34:52.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 10 librados: watch_flush exit 2024-09-10T20:34:52.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 20 client.54125.objecter shutdown clearing up homeless session... 2024-09-10T20:34:52.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 10 client.54125.objecter successfully canceled tick 2024-09-10T20:34:52.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3d0403cef0 msgr2=0x7f3d0403f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 --2- 172.21.15.153:0/3720235277 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3d0403cef0 0x7f3d0403f3b0 secure :-1 s=READY pgs=3373 cs=0 l=1 rev1=1 crypto rx=0x7f3d14004e30 tx=0x7f3d14009040 comp rx=0 tx=0).stop 2024-09-10T20:34:52.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 10 monclient: shutdown 2024-09-10T20:34:52.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 msgr2=0x7f3d2413f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 --2- 172.21.15.153:0/3720235277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413f2d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c0af040 tx=0x7f3d1c019690 comp rx=0 tx=0).stop 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 shutdown_connections 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 --2- 172.21.15.153:0/3720235277 >> v2:172.21.15.176:6812/3003473398 conn(0x7f3d0403cef0 0x7f3d0403f3b0 unknown :-1 s=CLOSED pgs=3373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 --2- 172.21.15.153:0/3720235277 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f3d240a49d0 0x7f3d2413f2d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 --2- 172.21.15.153:0/3720235277 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f3d240a4580 0x7f3d24126300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 --2- 172.21.15.153:0/3720235277 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f3d240a41a0 0x7f3d24128460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 >> 172.21.15.153:0/3720235277 conn(0x7f3d240ab7e0 msgr2=0x7f3d240a1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 shutdown_connections 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 -- 172.21.15.153:0/3720235277 wait complete. 2024-09-10T20:34:52.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.533+0000 7f3d29217640 1 librados: shutdown 2024-09-10T20:34:52.552 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":571,"stamp":"2024-09-10T20:34:51.249593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":449376,"kb_used_data":28984,"kb_used_omap":9,"kb_used_meta":420214,"kb_avail":565781664,"statfs":{"total":579820584960,"available":579360423936,"internally_reserved":0,"allocated":29679616,"data_stored":15420662,"data_compressed":596482,"data_compressed_allocated":966656,"data_compressed_original":3112960,"omap_allocated":9567,"internal_metadata":430299809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003835"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.828961+0000","last_change":"2024-09-10T20:27:17.198257+0000","last_active":"2024-09-10T20:33:29.828961+0000","last_peered":"2024-09-10T20:33:29.828961+0000","last_clean":"2024-09-10T20:33:29.828961+0000","last_became_active":"2024-09-10T20:27:17.197839+0000","last_became_peered":"2024-09-10T20:27:17.197839+0000","last_unstale":"2024-09-10T20:33:29.828961+0000","last_undegraded":"2024-09-10T20:33:29.828961+0000","last_fullsized":"2024-09-10T20:33:29.828961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:15.959864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500272+0000","last_change":"2024-09-10T20:27:17.223318+0000","last_active":"2024-09-10T20:33:29.500272+0000","last_peered":"2024-09-10T20:33:29.500272+0000","last_clean":"2024-09-10T20:33:29.500272+0000","last_became_active":"2024-09-10T20:27:17.219498+0000","last_became_peered":"2024-09-10T20:27:17.219498+0000","last_unstale":"2024-09-10T20:33:29.500272+0000","last_undegraded":"2024-09-10T20:33:29.500272+0000","last_fullsized":"2024-09-10T20:33:29.500272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:12.081089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498568+0000","last_change":"2024-09-10T20:27:17.202919+0000","last_active":"2024-09-10T20:33:29.498568+0000","last_peered":"2024-09-10T20:33:29.498568+0000","last_clean":"2024-09-10T20:33:29.498568+0000","last_became_active":"2024-09-10T20:27:17.202373+0000","last_became_peered":"2024-09-10T20:27:17.202373+0000","last_unstale":"2024-09-10T20:33:29.498568+0000","last_undegraded":"2024-09-10T20:33:29.498568+0000","last_fullsized":"2024-09-10T20:33:29.498568+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:42:27.202536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500829+0000","last_change":"2024-09-10T20:27:17.218042+0000","last_active":"2024-09-10T20:33:29.500829+0000","last_peered":"2024-09-10T20:33:29.500829+0000","last_clean":"2024-09-10T20:33:29.500829+0000","last_became_active":"2024-09-10T20:27:17.217517+0000","last_became_peered":"2024-09-10T20:27:17.217517+0000","last_unstale":"2024-09-10T20:33:29.500829+0000","last_undegraded":"2024-09-10T20:33:29.500829+0000","last_fullsized":"2024-09-10T20:33:29.500829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:53.695011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829236+0000","last_change":"2024-09-10T20:27:17.282375+0000","last_active":"2024-09-10T20:33:29.829236+0000","last_peered":"2024-09-10T20:33:29.829236+0000","last_clean":"2024-09-10T20:33:29.829236+0000","last_became_active":"2024-09-10T20:27:17.281836+0000","last_became_peered":"2024-09-10T20:27:17.281836+0000","last_unstale":"2024-09-10T20:33:29.829236+0000","last_undegraded":"2024-09-10T20:33:29.829236+0000","last_fullsized":"2024-09-10T20:33:29.829236+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:51:27.168991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498729+0000","last_change":"2024-09-10T20:27:17.213190+0000","last_active":"2024-09-10T20:33:29.498729+0000","last_peered":"2024-09-10T20:33:29.498729+0000","last_clean":"2024-09-10T20:33:29.498729+0000","last_became_active":"2024-09-10T20:27:17.212620+0000","last_became_peered":"2024-09-10T20:27:17.212620+0000","last_unstale":"2024-09-10T20:33:29.498729+0000","last_undegraded":"2024-09-10T20:33:29.498729+0000","last_fullsized":"2024-09-10T20:33:29.498729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:48.977530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498751+0000","last_change":"2024-09-10T20:27:17.288663+0000","last_active":"2024-09-10T20:33:29.498751+0000","last_peered":"2024-09-10T20:33:29.498751+0000","last_clean":"2024-09-10T20:33:29.498751+0000","last_became_active":"2024-09-10T20:27:17.287973+0000","last_became_peered":"2024-09-10T20:27:17.287973+0000","last_unstale":"2024-09-10T20:33:29.498751+0000","last_undegraded":"2024-09-10T20:33:29.498751+0000","last_fullsized":"2024-09-10T20:33:29.498751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:16:25.413353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874217+0000","last_change":"2024-09-10T20:27:17.210079+0000","last_active":"2024-09-10T20:33:29.874217+0000","last_peered":"2024-09-10T20:33:29.874217+0000","last_clean":"2024-09-10T20:33:29.874217+0000","last_became_active":"2024-09-10T20:27:17.209693+0000","last_became_peered":"2024-09-10T20:27:17.209693+0000","last_unstale":"2024-09-10T20:33:29.874217+0000","last_undegraded":"2024-09-10T20:33:29.874217+0000","last_fullsized":"2024-09-10T20:33:29.874217+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:44.983674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500800+0000","last_change":"2024-09-10T20:27:17.249972+0000","last_active":"2024-09-10T20:33:29.500800+0000","last_peered":"2024-09-10T20:33:29.500800+0000","last_clean":"2024-09-10T20:33:29.500800+0000","last_became_active":"2024-09-10T20:27:17.249683+0000","last_became_peered":"2024-09-10T20:27:17.249683+0000","last_unstale":"2024-09-10T20:33:29.500800+0000","last_undegraded":"2024-09-10T20:33:29.500800+0000","last_fullsized":"2024-09-10T20:33:29.500800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:02:11.684811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498432+0000","last_change":"2024-09-10T20:27:17.248264+0000","last_active":"2024-09-10T20:33:29.498432+0000","last_peered":"2024-09-10T20:33:29.498432+0000","last_clean":"2024-09-10T20:33:29.498432+0000","last_became_active":"2024-09-10T20:27:17.247865+0000","last_became_peered":"2024-09-10T20:27:17.247865+0000","last_unstale":"2024-09-10T20:33:29.498432+0000","last_undegraded":"2024-09-10T20:33:29.498432+0000","last_fullsized":"2024-09-10T20:33:29.498432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:06:33.643650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829007+0000","last_change":"2024-09-10T20:27:17.203381+0000","last_active":"2024-09-10T20:33:29.829007+0000","last_peered":"2024-09-10T20:33:29.829007+0000","last_clean":"2024-09-10T20:33:29.829007+0000","last_became_active":"2024-09-10T20:27:17.203002+0000","last_became_peered":"2024-09-10T20:27:17.203002+0000","last_unstale":"2024-09-10T20:33:29.829007+0000","last_undegraded":"2024-09-10T20:33:29.829007+0000","last_fullsized":"2024-09-10T20:33:29.829007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:32:20.452158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873964+0000","last_change":"2024-09-10T20:27:17.298627+0000","last_active":"2024-09-10T20:33:29.873964+0000","last_peered":"2024-09-10T20:33:29.873964+0000","last_clean":"2024-09-10T20:33:29.873964+0000","last_became_active":"2024-09-10T20:27:17.298309+0000","last_became_peered":"2024-09-10T20:27:17.298309+0000","last_unstale":"2024-09-10T20:33:29.873964+0000","last_undegraded":"2024-09-10T20:33:29.873964+0000","last_fullsized":"2024-09-10T20:33:29.873964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:55.361013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":596,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500685+0000","last_change":"2024-09-10T20:26:21.773251+0000","last_active":"2024-09-10T20:33:29.500685+0000","last_peered":"2024-09-10T20:33:29.500685+0000","last_clean":"2024-09-10T20:33:29.500685+0000","last_became_active":"2024-09-10T20:26:19.356844+0000","last_became_peered":"2024-09-10T20:26:19.356844+0000","last_unstale":"2024-09-10T20:33:29.500685+0000","last_undegraded":"2024-09-10T20:33:29.500685+0000","last_fullsized":"2024-09-10T20:33:29.500685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:23.413224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873697+0000","last_change":"2024-09-10T20:27:17.206032+0000","last_active":"2024-09-10T20:33:29.873697+0000","last_peered":"2024-09-10T20:33:29.873697+0000","last_clean":"2024-09-10T20:33:29.873697+0000","last_became_active":"2024-09-10T20:27:17.205464+0000","last_became_peered":"2024-09-10T20:27:17.205464+0000","last_unstale":"2024-09-10T20:33:29.873697+0000","last_undegraded":"2024-09-10T20:33:29.873697+0000","last_fullsized":"2024-09-10T20:33:29.873697+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:33.219827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":593,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829238+0000","last_change":"2024-09-10T20:26:21.381255+0000","last_active":"2024-09-10T20:33:29.829238+0000","last_peered":"2024-09-10T20:33:29.829238+0000","last_clean":"2024-09-10T20:33:29.829238+0000","last_became_active":"2024-09-10T20:26:19.363220+0000","last_became_peered":"2024-09-10T20:26:19.363220+0000","last_unstale":"2024-09-10T20:33:29.829238+0000","last_undegraded":"2024-09-10T20:33:29.829238+0000","last_fullsized":"2024-09-10T20:33:29.829238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:09.279291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096679600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"23'7","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500753+0000","last_change":"2024-09-10T20:27:17.217628+0000","last_active":"2024-09-10T20:33:29.500753+0000","last_peered":"2024-09-10T20:33:29.500753+0000","last_clean":"2024-09-10T20:33:29.500753+0000","last_became_active":"2024-09-10T20:27:17.211037+0000","last_became_peered":"2024-09-10T20:27:17.211037+0000","last_unstale":"2024-09-10T20:33:29.500753+0000","last_undegraded":"2024-09-10T20:33:29.500753+0000","last_fullsized":"2024-09-10T20:33:29.500753+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:47.151425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":597,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500610+0000","last_change":"2024-09-10T20:26:21.773020+0000","last_active":"2024-09-10T20:33:29.500610+0000","last_peered":"2024-09-10T20:33:29.500610+0000","last_clean":"2024-09-10T20:33:29.500610+0000","last_became_active":"2024-09-10T20:26:19.363294+0000","last_became_peered":"2024-09-10T20:26:19.363294+0000","last_unstale":"2024-09-10T20:33:29.500610+0000","last_undegraded":"2024-09-10T20:33:29.500610+0000","last_fullsized":"2024-09-10T20:33:29.500610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:54.300619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053952999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.875890+0000","last_change":"2024-09-10T20:27:17.298070+0000","last_active":"2024-09-10T20:33:29.875890+0000","last_peered":"2024-09-10T20:33:29.875890+0000","last_clean":"2024-09-10T20:33:29.875890+0000","last_became_active":"2024-09-10T20:27:17.297745+0000","last_became_peered":"2024-09-10T20:27:17.297745+0000","last_unstale":"2024-09-10T20:33:29.875890+0000","last_undegraded":"2024-09-10T20:33:29.875890+0000","last_fullsized":"2024-09-10T20:33:29.875890+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:30:22.598793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874003+0000","last_change":"2024-09-10T20:26:21.378377+0000","last_active":"2024-09-10T20:33:29.874003+0000","last_peered":"2024-09-10T20:33:29.874003+0000","last_clean":"2024-09-10T20:33:29.874003+0000","last_became_active":"2024-09-10T20:26:19.359961+0000","last_became_peered":"2024-09-10T20:26:19.359961+0000","last_unstale":"2024-09-10T20:33:29.874003+0000","last_undegraded":"2024-09-10T20:33:29.874003+0000","last_fullsized":"2024-09-10T20:33:29.874003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:21:01.940376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829065+0000","last_change":"2024-09-10T20:27:17.282394+0000","last_active":"2024-09-10T20:33:29.829065+0000","last_peered":"2024-09-10T20:33:29.829065+0000","last_clean":"2024-09-10T20:33:29.829065+0000","last_became_active":"2024-09-10T20:27:17.281815+0000","last_became_peered":"2024-09-10T20:27:17.281815+0000","last_unstale":"2024-09-10T20:33:29.829065+0000","last_undegraded":"2024-09-10T20:33:29.829065+0000","last_fullsized":"2024-09-10T20:33:29.829065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:49.830688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":597,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876148+0000","last_change":"2024-09-10T20:26:21.386697+0000","last_active":"2024-09-10T20:33:29.876148+0000","last_peered":"2024-09-10T20:33:29.876148+0000","last_clean":"2024-09-10T20:33:29.876148+0000","last_became_active":"2024-09-10T20:26:19.360427+0000","last_became_peered":"2024-09-10T20:26:19.360427+0000","last_unstale":"2024-09-10T20:33:29.876148+0000","last_undegraded":"2024-09-10T20:33:29.876148+0000","last_fullsized":"2024-09-10T20:33:29.876148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:53.136006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020581079999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873800+0000","last_change":"2024-09-10T20:27:17.282845+0000","last_active":"2024-09-10T20:33:29.873800+0000","last_peered":"2024-09-10T20:33:29.873800+0000","last_clean":"2024-09-10T20:33:29.873800+0000","last_became_active":"2024-09-10T20:27:17.282600+0000","last_became_peered":"2024-09-10T20:27:17.282600+0000","last_unstale":"2024-09-10T20:33:29.873800+0000","last_undegraded":"2024-09-10T20:33:29.873800+0000","last_fullsized":"2024-09-10T20:33:29.873800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:38.719966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":603,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873765+0000","last_change":"2024-09-10T20:26:21.575187+0000","last_active":"2024-09-10T20:33:29.873765+0000","last_peered":"2024-09-10T20:33:29.873765+0000","last_clean":"2024-09-10T20:33:29.873765+0000","last_became_active":"2024-09-10T20:26:19.362858+0000","last_became_peered":"2024-09-10T20:26:19.362858+0000","last_unstale":"2024-09-10T20:33:29.873765+0000","last_undegraded":"2024-09-10T20:33:29.873765+0000","last_fullsized":"2024-09-10T20:33:29.873765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:22:17.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078714099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829346+0000","last_change":"2024-09-10T20:27:17.214784+0000","last_active":"2024-09-10T20:33:29.829346+0000","last_peered":"2024-09-10T20:33:29.829346+0000","last_clean":"2024-09-10T20:33:29.829346+0000","last_became_active":"2024-09-10T20:27:17.214428+0000","last_became_peered":"2024-09-10T20:27:17.214428+0000","last_unstale":"2024-09-10T20:33:29.829346+0000","last_undegraded":"2024-09-10T20:33:29.829346+0000","last_fullsized":"2024-09-10T20:33:29.829346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:50:15.983575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":610,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498452+0000","last_change":"2024-09-10T20:26:21.378915+0000","last_active":"2024-09-10T20:33:29.498452+0000","last_peered":"2024-09-10T20:33:29.498452+0000","last_clean":"2024-09-10T20:33:29.498452+0000","last_became_active":"2024-09-10T20:26:19.362158+0000","last_became_peered":"2024-09-10T20:26:19.362158+0000","last_unstale":"2024-09-10T20:33:29.498452+0000","last_undegraded":"2024-09-10T20:33:29.498452+0000","last_fullsized":"2024-09-10T20:33:29.498452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:50.708554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010122729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876226+0000","last_change":"2024-09-10T20:27:17.244860+0000","last_active":"2024-09-10T20:33:29.876226+0000","last_peered":"2024-09-10T20:33:29.876226+0000","last_clean":"2024-09-10T20:33:29.876226+0000","last_became_active":"2024-09-10T20:27:17.243935+0000","last_became_peered":"2024-09-10T20:27:17.243935+0000","last_unstale":"2024-09-10T20:33:29.876226+0000","last_undegraded":"2024-09-10T20:33:29.876226+0000","last_fullsized":"2024-09-10T20:33:29.876226+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:11.085907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":598,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500566+0000","last_change":"2024-09-10T20:26:21.773025+0000","last_active":"2024-09-10T20:33:29.500566+0000","last_peered":"2024-09-10T20:33:29.500566+0000","last_clean":"2024-09-10T20:33:29.500566+0000","last_became_active":"2024-09-10T20:26:19.359775+0000","last_became_peered":"2024-09-10T20:26:19.359775+0000","last_unstale":"2024-09-10T20:33:29.500566+0000","last_undegraded":"2024-09-10T20:33:29.500566+0000","last_fullsized":"2024-09-10T20:33:29.500566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:35.427907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054594699999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":688,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500496+0000","last_change":"2024-09-10T20:26:14.579551+0000","last_active":"2024-09-10T20:33:29.500496+0000","last_peered":"2024-09-10T20:33:29.500496+0000","last_clean":"2024-09-10T20:33:29.500496+0000","last_became_active":"2024-09-10T20:26:14.577602+0000","last_became_peered":"2024-09-10T20:26:14.577602+0000","last_unstale":"2024-09-10T20:33:29.500496+0000","last_undegraded":"2024-09-10T20:33:29.500496+0000","last_fullsized":"2024-09-10T20:33:29.500496+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_clean_scrub_stamp":"2024-09-10T20:26:13.287430+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:41.463707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.875916+0000","last_change":"2024-09-10T20:27:17.297570+0000","last_active":"2024-09-10T20:33:29.875916+0000","last_peered":"2024-09-10T20:33:29.875916+0000","last_clean":"2024-09-10T20:33:29.875916+0000","last_became_active":"2024-09-10T20:27:17.295709+0000","last_became_peered":"2024-09-10T20:27:17.295709+0000","last_unstale":"2024-09-10T20:33:29.875916+0000","last_undegraded":"2024-09-10T20:33:29.875916+0000","last_fullsized":"2024-09-10T20:33:29.875916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:37.519789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498888+0000","last_change":"2024-09-10T20:27:17.219710+0000","last_active":"2024-09-10T20:33:29.498888+0000","last_peered":"2024-09-10T20:33:29.498888+0000","last_clean":"2024-09-10T20:33:29.498888+0000","last_became_active":"2024-09-10T20:27:17.217973+0000","last_became_peered":"2024-09-10T20:27:17.217973+0000","last_unstale":"2024-09-10T20:33:29.498888+0000","last_undegraded":"2024-09-10T20:33:29.498888+0000","last_fullsized":"2024-09-10T20:33:29.498888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:55.139486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500679+0000","last_change":"2024-09-10T20:27:17.318458+0000","last_active":"2024-09-10T20:33:29.500679+0000","last_peered":"2024-09-10T20:33:29.500679+0000","last_clean":"2024-09-10T20:33:29.500679+0000","last_became_active":"2024-09-10T20:27:17.318202+0000","last_became_peered":"2024-09-10T20:27:17.318202+0000","last_unstale":"2024-09-10T20:33:29.500679+0000","last_undegraded":"2024-09-10T20:33:29.500679+0000","last_fullsized":"2024-09-10T20:33:29.500679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:49.111756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873865+0000","last_change":"2024-09-10T20:27:17.240732+0000","last_active":"2024-09-10T20:33:29.873865+0000","last_peered":"2024-09-10T20:33:29.873865+0000","last_clean":"2024-09-10T20:33:29.873865+0000","last_became_active":"2024-09-10T20:27:17.240435+0000","last_became_peered":"2024-09-10T20:27:17.240435+0000","last_unstale":"2024-09-10T20:33:29.873865+0000","last_undegraded":"2024-09-10T20:33:29.873865+0000","last_fullsized":"2024-09-10T20:33:29.873865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:54:42.900319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874276+0000","last_change":"2024-09-10T20:27:17.230157+0000","last_active":"2024-09-10T20:33:29.874276+0000","last_peered":"2024-09-10T20:33:29.874276+0000","last_clean":"2024-09-10T20:33:29.874276+0000","last_became_active":"2024-09-10T20:27:17.229886+0000","last_became_peered":"2024-09-10T20:27:17.229886+0000","last_unstale":"2024-09-10T20:33:29.874276+0000","last_undegraded":"2024-09-10T20:33:29.874276+0000","last_fullsized":"2024-09-10T20:33:29.874276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:40.265668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874302+0000","last_change":"2024-09-10T20:27:17.214355+0000","last_active":"2024-09-10T20:33:29.874302+0000","last_peered":"2024-09-10T20:33:29.874302+0000","last_clean":"2024-09-10T20:33:29.874302+0000","last_became_active":"2024-09-10T20:27:17.213875+0000","last_became_peered":"2024-09-10T20:27:17.213875+0000","last_unstale":"2024-09-10T20:33:29.874302+0000","last_undegraded":"2024-09-10T20:33:29.874302+0000","last_fullsized":"2024-09-10T20:33:29.874302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:55.411553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873798+0000","last_change":"2024-09-10T20:27:17.288406+0000","last_active":"2024-09-10T20:33:29.873798+0000","last_peered":"2024-09-10T20:33:29.873798+0000","last_clean":"2024-09-10T20:33:29.873798+0000","last_became_active":"2024-09-10T20:27:17.288092+0000","last_became_peered":"2024-09-10T20:27:17.288092+0000","last_unstale":"2024-09-10T20:33:29.873798+0000","last_undegraded":"2024-09-10T20:33:29.873798+0000","last_fullsized":"2024-09-10T20:33:29.873798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:47.584995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873374+0000","last_change":"2024-09-10T20:27:17.210249+0000","last_active":"2024-09-10T20:33:29.873374+0000","last_peered":"2024-09-10T20:33:29.873374+0000","last_clean":"2024-09-10T20:33:29.873374+0000","last_became_active":"2024-09-10T20:27:17.209363+0000","last_became_peered":"2024-09-10T20:27:17.209363+0000","last_unstale":"2024-09-10T20:33:29.873374+0000","last_undegraded":"2024-09-10T20:33:29.873374+0000","last_fullsized":"2024-09-10T20:33:29.873374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:19.942713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500468+0000","last_change":"2024-09-10T20:27:17.225434+0000","last_active":"2024-09-10T20:33:29.500468+0000","last_peered":"2024-09-10T20:33:29.500468+0000","last_clean":"2024-09-10T20:33:29.500468+0000","last_became_active":"2024-09-10T20:27:17.224230+0000","last_became_peered":"2024-09-10T20:27:17.224230+0000","last_unstale":"2024-09-10T20:33:29.500468+0000","last_undegraded":"2024-09-10T20:33:29.500468+0000","last_fullsized":"2024-09-10T20:33:29.500468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:49:09.800954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500670+0000","last_change":"2024-09-10T20:27:17.217944+0000","last_active":"2024-09-10T20:33:29.500670+0000","last_peered":"2024-09-10T20:33:29.500670+0000","last_clean":"2024-09-10T20:33:29.500670+0000","last_became_active":"2024-09-10T20:27:17.217107+0000","last_became_peered":"2024-09-10T20:27:17.217107+0000","last_unstale":"2024-09-10T20:33:29.500670+0000","last_undegraded":"2024-09-10T20:33:29.500670+0000","last_fullsized":"2024-09-10T20:33:29.500670+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:52.708045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829272+0000","last_change":"2024-09-10T20:27:17.283023+0000","last_active":"2024-09-10T20:33:29.829272+0000","last_peered":"2024-09-10T20:33:29.829272+0000","last_clean":"2024-09-10T20:33:29.829272+0000","last_became_active":"2024-09-10T20:27:17.282074+0000","last_became_peered":"2024-09-10T20:27:17.282074+0000","last_unstale":"2024-09-10T20:33:29.829272+0000","last_undegraded":"2024-09-10T20:33:29.829272+0000","last_fullsized":"2024-09-10T20:33:29.829272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:51.986119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874322+0000","last_change":"2024-09-10T20:27:17.211220+0000","last_active":"2024-09-10T20:33:29.874322+0000","last_peered":"2024-09-10T20:33:29.874322+0000","last_clean":"2024-09-10T20:33:29.874322+0000","last_became_active":"2024-09-10T20:27:17.210803+0000","last_became_peered":"2024-09-10T20:27:17.210803+0000","last_unstale":"2024-09-10T20:33:29.874322+0000","last_undegraded":"2024-09-10T20:33:29.874322+0000","last_fullsized":"2024-09-10T20:33:29.874322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:38.315319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876046+0000","last_change":"2024-09-10T20:27:17.311360+0000","last_active":"2024-09-10T20:33:29.876046+0000","last_peered":"2024-09-10T20:33:29.876046+0000","last_clean":"2024-09-10T20:33:29.876046+0000","last_became_active":"2024-09-10T20:27:17.310182+0000","last_became_peered":"2024-09-10T20:27:17.310182+0000","last_unstale":"2024-09-10T20:33:29.876046+0000","last_undegraded":"2024-09-10T20:33:29.876046+0000","last_fullsized":"2024-09-10T20:33:29.876046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:34:02.949919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574956,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80416,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291424,"statfs":{"total":96636764160,"available":96554418176,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80488,"kb_used_data":4872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291352,"statfs":{"total":96636764160,"available":96554344448,"internally_reserved":0,"allocated":4988928,"data_stored":2922726,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574953,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76260,"kb_used_data":4812,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295580,"statfs":{"total":96636764160,"available":96558673920,"internally_reserved":0,"allocated":4927488,"data_stored":2393812,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640363,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72172,"kb_used_data":4868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299668,"statfs":{"total":96636764160,"available":96562860032,"internally_reserved":0,"allocated":4984832,"data_stored":2922721,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67944,"kb_used_data":4816,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303896,"statfs":{"total":96636764160,"available":96567189504,"internally_reserved":0,"allocated":4931584,"data_stored":2393817,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607658,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72096,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299744,"statfs":{"total":96636764160,"available":96562937856,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:34:52.555 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-10T20:34:52.555 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.665+0000 7fcc64bae640 10 monclient: get_monmap_and_config 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.665+0000 7fcc64bae640 10 monclient: build_initial_monmap 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.665+0000 7fcc64bae640 10 monclient: monmap: 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:52.668900+0000 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:52.668900+0000 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:52.669 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:52.670 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:52.670 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.665+0000 7fcc64bae640 1 Processor -- start 2024-09-10T20:34:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.665+0000 7fcc64bae640 1 -- start start 2024-09-10T20:34:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.665+0000 7fcc64bae640 10 monclient: init 2024-09-10T20:34:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 10 monclient: _add_conns ranks=[3,0,4,2,1] 2024-09-10T20:34:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcc600a5ab0 0x7fcc600a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.672 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 10 monclient(hunting): picked mon.noname-a con 0x7fcc600a5ab0 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:52.674 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fcc600a63d0 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fcc601390d0 0x7fcc60140c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 10 monclient(hunting): picked mon.noname-c con 0x7fcc601390d0 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc60070780 con 0x7fcc600a63d0 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fcc600a3010 con 0x7fcc600a5ab0 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fcc600709c0 con 0x7fcc601390d0 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc64bae640 10 monclient(hunting): authenticate will time out at 1452.808479s 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fcc600a5ab0 0x7fcc600a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5dd9b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fcc600a63d0 0x7fcc601389c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60718/0 (socket says 172.21.15.153:60718) 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5dd9b640 1 -- 172.21.15.153:0/1757121495 learned_addr learned my addr 172.21.15.153:0/1757121495 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:52.675 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient(hunting): get_auth_request con 0x7fcc600a5ab0 auth_method 0 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5ed9d640 1 --2- 172.21.15.153:0/1757121495 >> v2:172.21.15.176:3300/0 conn(0x7fcc601390d0 0x7fcc60140c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient(hunting): handle_auth_done global_id 34389 payload 274 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 1 -- 172.21.15.153:0/1757121495 >> v2:172.21.15.176:3300/0 conn(0x7fcc601390d0 msgr2=0x7fcc60140c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 1 --2- 172.21.15.153:0/1757121495 >> v2:172.21.15.176:3300/0 conn(0x7fcc601390d0 0x7fcc60140c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 1 -- 172.21.15.153:0/1757121495 >> v1:172.21.15.153:6789/0 conn(0x7fcc600a63d0 legacy=0x7fcc601389c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:52.676 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient: _finish_hunting 0 2024-09-10T20:34:52.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 1 monclient: found mon.noname-a 2024-09-10T20:34:52.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:52.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:52.677 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.669+0000 7fcc5e59c640 1 -- 172.21.15.153:0/1757121495 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc600a26e0 con 0x7fcc600a5ab0 2024-09-10T20:34:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5e59c640 1 -- 172.21.15.153:0/1757121495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 msgr2=0x7fcc600a5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5e59c640 1 --2- 172.21.15.153:0/1757121495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 0x7fcc600a5e90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc600a3430 tx=0x7fcc5801c7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=488ef1479ecaa631 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 1 -- 172.21.15.153:0/1757121495 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc580b2070 con 0x7fcc600a5ab0 2024-09-10T20:34:52.683 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 10 monclient: dump: 2024-09-10T20:34:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.684 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.685 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 10 monclient: _finish_auth 0 2024-09-10T20:34:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 10 monclient: _check_auth_tickets 2024-09-10T20:34:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:52.685 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 5 monclient: authenticate success, global_id 34389 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 1 -- 172.21.15.153:0/1757121495 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc580a5970 con 0x7fcc600a5ab0 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 4 set_mon_vals no callback set 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 1 -- 172.21.15.153:0/1757121495 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc580a5d40 con 0x7fcc600a5ab0 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5d59a640 10 monclient: dump: 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.686 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient: shutdown 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 -- 172.21.15.153:0/1757121495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 msgr2=0x7fcc600a5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 --2- 172.21.15.153:0/1757121495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 0x7fcc600a5e90 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcc600a3430 tx=0x7fcc5801c7f0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.687 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 -- 172.21.15.153:0/1757121495 shutdown_connections 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 --2- 172.21.15.153:0/1757121495 >> v2:172.21.15.176:3300/0 conn(0x7fcc601390d0 0x7fcc60140c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 --2- 172.21.15.153:0/1757121495 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 0x7fcc600a5e90 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 -- 172.21.15.153:0/1757121495 >> 172.21.15.153:0/1757121495 conn(0x7fcc600aba00 msgr2=0x7fcc600a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 -- 172.21.15.153:0/1757121495 shutdown_connections 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 -- 172.21.15.153:0/1757121495 wait complete. 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient: build_initial_monmap 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient: monmap: 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:52.688 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:52.689 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.689 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:52.689 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:52.689 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:52.690 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 librados: starting msgr at 2024-09-10T20:34:52.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 librados: starting objecter 2024-09-10T20:34:52.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 Processor -- start 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 -- start start 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 librados: setting wanted keys 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 librados: calling monclient init 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient: init 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 0x7fcc601d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient(hunting): picked mon.noname-a con 0x7fcc600a5ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc600a63d0 0x7fcc60137a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.691 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient(hunting): picked mon.noname-c con 0x7fcc600a63d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:52.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc601390d0 0x7fcc6013db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient(hunting): picked mon.noname-b con 0x7fcc601390d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:52.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fcc600a3960 con 0x7fcc600a5ab0 2024-09-10T20:34:52.692 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fcc600a3ae0 con 0x7fcc601390d0 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fcc601439f0 con 0x7fcc600a63d0 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc64bae640 10 monclient(hunting): authenticate will time out at 1452.815820s 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 0x7fcc601d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5dd9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc600a63d0 0x7fcc60137a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5e59c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 0x7fcc601d0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56190/0 (socket says 172.21.15.153:56190) 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5e59c640 1 -- 172.21.15.153:0/1689680841 learned_addr learned my addr 172.21.15.153:0/1689680841 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:52.693 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5e59c640 10 monclient(hunting): get_auth_request con 0x7fcc600a5ab0 auth_method 0 2024-09-10T20:34:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5dd9b640 10 monclient(hunting): get_auth_request con 0x7fcc600a63d0 auth_method 0 2024-09-10T20:34:52.694 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.673+0000 7fcc5dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5dd9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5dd9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5dd9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 10 monclient(hunting): handle_auth_done global_id 34392 payload 824 2024-09-10T20:34:52.695 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5ed9d640 1 --2- 172.21.15.153:0/1689680841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc601390d0 0x7fcc6013db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 1 -- 172.21.15.153:0/1689680841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc600a63d0 msgr2=0x7fcc60137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 1 --2- 172.21.15.153:0/1689680841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc600a63d0 0x7fcc60137a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 1 -- 172.21.15.153:0/1689680841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc601390d0 msgr2=0x7fcc6013db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 1 --2- 172.21.15.153:0/1689680841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc601390d0 0x7fcc6013db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 10 monclient: _finish_hunting 0 2024-09-10T20:34:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 1 monclient: found mon.noname-a 2024-09-10T20:34:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:52.696 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 1 -- 172.21.15.153:0/1689680841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc60121700 con 0x7fcc600a5ab0 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5ed9d640 10 monclient: get_auth_request con 0x7fcc601390d0 auth_method 0 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5ed9d640 1 --2- 172.21.15.153:0/1689680841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc601390d0 0x7fcc6013db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 1 -- 172.21.15.153:0/1689680841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 msgr2=0x7fcc601d0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5e59c640 1 --2- 172.21.15.153:0/1689680841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 0x7fcc601d0b00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcc5801aea0 tx=0x7fcc58019590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 client.?.objecter ms_handle_connect 0x7fcc600a5ab0 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 1 -- 172.21.15.153:0/1689680841 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc580bd020 con 0x7fcc600a5ab0 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 monclient: dump: 2024-09-10T20:34:52.697 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 monclient: _finish_auth 0 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:52.698 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 1 -- 172.21.15.153:0/1689680841 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc580b8880 con 0x7fcc600a5ab0 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 5 monclient: authenticate success, global_id 34392 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 10 monclient: _renew_subs 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc600a22a0 con 0x7fcc600a5ab0 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 1 -- 172.21.15.153:0/1689680841 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc580b2040 con 0x7fcc600a5ab0 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 monclient: dump: 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:52.699 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc5cd99640 4 set_mon_vals no callback set 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 10 client.34392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 10 monclient: _renew_subs 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc601223b0 con 0x7fcc600a5ab0 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 1 librados: init done 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 1 -- 172.21.15.153:0/1689680841 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fcc580c0020 con 0x7fcc600a5ab0 2024-09-10T20:34:52.700 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc600a5e90 con 0x7fcc600a5ab0 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 1 --2- 172.21.15.153:0/1689680841 >> v2:172.21.15.176:6812/3003473398 conn(0x7fcc4003cea0 0x7fcc4003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 1 -- 172.21.15.153:0/1689680841 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7fcc580ea510 con 0x7fcc600a5ab0 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 10 client.34392.objecter ms_dispatch 0x7fcc60000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 3 client.34392.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 3 client.34392.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.677+0000 7fcc56ffd640 20 client.34392.objecter dump_active .. 0 homeless 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.681+0000 7fcc5dd9b640 1 --2- 172.21.15.153:0/1689680841 >> v2:172.21.15.176:6812/3003473398 conn(0x7fcc4003cea0 0x7fcc4003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:52.701 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.681+0000 7fcc5dd9b640 10 monclient: get_auth_request con 0x7fcc4003cea0 auth_method 0 2024-09-10T20:34:52.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.681+0000 7fcc56ffd640 10 client.34392.objecter ms_handle_connect 0x7fcc4003cea0 2024-09-10T20:34:52.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.681+0000 7fcc5dd9b640 1 --2- 172.21.15.153:0/1689680841 >> v2:172.21.15.176:6812/3003473398 conn(0x7fcc4003cea0 0x7fcc4003f360 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7fcc48007ca0 tx=0x7fcc48007bf0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:52.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.685+0000 7fcc56ffd640 1 -- 172.21.15.153:0/1689680841 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fcc580c7070 con 0x7fcc600a5ab0 2024-09-10T20:34:52.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.685+0000 7fcc56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:52.702 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.685+0000 7fcc56ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:52.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.849+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcc6013f610 con 0x7fcc4003cea0 2024-09-10T20:34:52.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.853+0000 7fcc56ffd640 1 -- 172.21.15.153:0/1689680841 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114145 (secure 0 0 0) 0x7fcc6013f610 con 0x7fcc4003cea0 2024-09-10T20:34:52.859 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:34:52.859 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-10T20:34:52.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 10 librados: watch_flush enter 2024-09-10T20:34:52.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 10 librados: watch_flush exit 2024-09-10T20:34:52.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 20 client.34392.objecter shutdown clearing up homeless session... 2024-09-10T20:34:52.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 10 client.34392.objecter successfully canceled tick 2024-09-10T20:34:52.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 >> v2:172.21.15.176:6812/3003473398 conn(0x7fcc4003cea0 msgr2=0x7fcc4003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 --2- 172.21.15.153:0/1689680841 >> v2:172.21.15.176:6812/3003473398 conn(0x7fcc4003cea0 0x7fcc4003f360 secure :-1 s=READY pgs=3374 cs=0 l=1 rev1=1 crypto rx=0x7fcc48007ca0 tx=0x7fcc48007bf0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 10 monclient: shutdown 2024-09-10T20:34:52.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 msgr2=0x7fcc601d0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 --2- 172.21.15.153:0/1689680841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 0x7fcc601d0b00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcc5801aea0 tx=0x7fcc58019590 comp rx=0 tx=0).stop 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 shutdown_connections 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 --2- 172.21.15.153:0/1689680841 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fcc601390d0 0x7fcc6013db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 --2- 172.21.15.153:0/1689680841 >> v2:172.21.15.176:6812/3003473398 conn(0x7fcc4003cea0 0x7fcc4003f360 unknown :-1 s=CLOSED pgs=3374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 --2- 172.21.15.153:0/1689680841 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fcc600a63d0 0x7fcc60137a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 --2- 172.21.15.153:0/1689680841 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fcc600a5ab0 0x7fcc601d0b00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 >> 172.21.15.153:0/1689680841 conn(0x7fcc600aba00 msgr2=0x7fcc600aa090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 shutdown_connections 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 -- 172.21.15.153:0/1689680841 wait complete. 2024-09-10T20:34:52.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.857+0000 7fcc64bae640 1 librados: shutdown 2024-09-10T20:34:52.878 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":571,"stamp":"2024-09-10T20:34:51.249593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":449376,"kb_used_data":28984,"kb_used_omap":9,"kb_used_meta":420214,"kb_avail":565781664,"statfs":{"total":579820584960,"available":579360423936,"internally_reserved":0,"allocated":29679616,"data_stored":15420662,"data_compressed":596482,"data_compressed_allocated":966656,"data_compressed_original":3112960,"omap_allocated":9567,"internal_metadata":430299809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003835"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.828961+0000","last_change":"2024-09-10T20:27:17.198257+0000","last_active":"2024-09-10T20:33:29.828961+0000","last_peered":"2024-09-10T20:33:29.828961+0000","last_clean":"2024-09-10T20:33:29.828961+0000","last_became_active":"2024-09-10T20:27:17.197839+0000","last_became_peered":"2024-09-10T20:27:17.197839+0000","last_unstale":"2024-09-10T20:33:29.828961+0000","last_undegraded":"2024-09-10T20:33:29.828961+0000","last_fullsized":"2024-09-10T20:33:29.828961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:15.959864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500272+0000","last_change":"2024-09-10T20:27:17.223318+0000","last_active":"2024-09-10T20:33:29.500272+0000","last_peered":"2024-09-10T20:33:29.500272+0000","last_clean":"2024-09-10T20:33:29.500272+0000","last_became_active":"2024-09-10T20:27:17.219498+0000","last_became_peered":"2024-09-10T20:27:17.219498+0000","last_unstale":"2024-09-10T20:33:29.500272+0000","last_undegraded":"2024-09-10T20:33:29.500272+0000","last_fullsized":"2024-09-10T20:33:29.500272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:12.081089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498568+0000","last_change":"2024-09-10T20:27:17.202919+0000","last_active":"2024-09-10T20:33:29.498568+0000","last_peered":"2024-09-10T20:33:29.498568+0000","last_clean":"2024-09-10T20:33:29.498568+0000","last_became_active":"2024-09-10T20:27:17.202373+0000","last_became_peered":"2024-09-10T20:27:17.202373+0000","last_unstale":"2024-09-10T20:33:29.498568+0000","last_undegraded":"2024-09-10T20:33:29.498568+0000","last_fullsized":"2024-09-10T20:33:29.498568+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:42:27.202536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500829+0000","last_change":"2024-09-10T20:27:17.218042+0000","last_active":"2024-09-10T20:33:29.500829+0000","last_peered":"2024-09-10T20:33:29.500829+0000","last_clean":"2024-09-10T20:33:29.500829+0000","last_became_active":"2024-09-10T20:27:17.217517+0000","last_became_peered":"2024-09-10T20:27:17.217517+0000","last_unstale":"2024-09-10T20:33:29.500829+0000","last_undegraded":"2024-09-10T20:33:29.500829+0000","last_fullsized":"2024-09-10T20:33:29.500829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:53.695011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829236+0000","last_change":"2024-09-10T20:27:17.282375+0000","last_active":"2024-09-10T20:33:29.829236+0000","last_peered":"2024-09-10T20:33:29.829236+0000","last_clean":"2024-09-10T20:33:29.829236+0000","last_became_active":"2024-09-10T20:27:17.281836+0000","last_became_peered":"2024-09-10T20:27:17.281836+0000","last_unstale":"2024-09-10T20:33:29.829236+0000","last_undegraded":"2024-09-10T20:33:29.829236+0000","last_fullsized":"2024-09-10T20:33:29.829236+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:51:27.168991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498729+0000","last_change":"2024-09-10T20:27:17.213190+0000","last_active":"2024-09-10T20:33:29.498729+0000","last_peered":"2024-09-10T20:33:29.498729+0000","last_clean":"2024-09-10T20:33:29.498729+0000","last_became_active":"2024-09-10T20:27:17.212620+0000","last_became_peered":"2024-09-10T20:27:17.212620+0000","last_unstale":"2024-09-10T20:33:29.498729+0000","last_undegraded":"2024-09-10T20:33:29.498729+0000","last_fullsized":"2024-09-10T20:33:29.498729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:48.977530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498751+0000","last_change":"2024-09-10T20:27:17.288663+0000","last_active":"2024-09-10T20:33:29.498751+0000","last_peered":"2024-09-10T20:33:29.498751+0000","last_clean":"2024-09-10T20:33:29.498751+0000","last_became_active":"2024-09-10T20:27:17.287973+0000","last_became_peered":"2024-09-10T20:27:17.287973+0000","last_unstale":"2024-09-10T20:33:29.498751+0000","last_undegraded":"2024-09-10T20:33:29.498751+0000","last_fullsized":"2024-09-10T20:33:29.498751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:16:25.413353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874217+0000","last_change":"2024-09-10T20:27:17.210079+0000","last_active":"2024-09-10T20:33:29.874217+0000","last_peered":"2024-09-10T20:33:29.874217+0000","last_clean":"2024-09-10T20:33:29.874217+0000","last_became_active":"2024-09-10T20:27:17.209693+0000","last_became_peered":"2024-09-10T20:27:17.209693+0000","last_unstale":"2024-09-10T20:33:29.874217+0000","last_undegraded":"2024-09-10T20:33:29.874217+0000","last_fullsized":"2024-09-10T20:33:29.874217+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:44.983674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500800+0000","last_change":"2024-09-10T20:27:17.249972+0000","last_active":"2024-09-10T20:33:29.500800+0000","last_peered":"2024-09-10T20:33:29.500800+0000","last_clean":"2024-09-10T20:33:29.500800+0000","last_became_active":"2024-09-10T20:27:17.249683+0000","last_became_peered":"2024-09-10T20:27:17.249683+0000","last_unstale":"2024-09-10T20:33:29.500800+0000","last_undegraded":"2024-09-10T20:33:29.500800+0000","last_fullsized":"2024-09-10T20:33:29.500800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:02:11.684811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498432+0000","last_change":"2024-09-10T20:27:17.248264+0000","last_active":"2024-09-10T20:33:29.498432+0000","last_peered":"2024-09-10T20:33:29.498432+0000","last_clean":"2024-09-10T20:33:29.498432+0000","last_became_active":"2024-09-10T20:27:17.247865+0000","last_became_peered":"2024-09-10T20:27:17.247865+0000","last_unstale":"2024-09-10T20:33:29.498432+0000","last_undegraded":"2024-09-10T20:33:29.498432+0000","last_fullsized":"2024-09-10T20:33:29.498432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:06:33.643650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829007+0000","last_change":"2024-09-10T20:27:17.203381+0000","last_active":"2024-09-10T20:33:29.829007+0000","last_peered":"2024-09-10T20:33:29.829007+0000","last_clean":"2024-09-10T20:33:29.829007+0000","last_became_active":"2024-09-10T20:27:17.203002+0000","last_became_peered":"2024-09-10T20:27:17.203002+0000","last_unstale":"2024-09-10T20:33:29.829007+0000","last_undegraded":"2024-09-10T20:33:29.829007+0000","last_fullsized":"2024-09-10T20:33:29.829007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:32:20.452158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873964+0000","last_change":"2024-09-10T20:27:17.298627+0000","last_active":"2024-09-10T20:33:29.873964+0000","last_peered":"2024-09-10T20:33:29.873964+0000","last_clean":"2024-09-10T20:33:29.873964+0000","last_became_active":"2024-09-10T20:27:17.298309+0000","last_became_peered":"2024-09-10T20:27:17.298309+0000","last_unstale":"2024-09-10T20:33:29.873964+0000","last_undegraded":"2024-09-10T20:33:29.873964+0000","last_fullsized":"2024-09-10T20:33:29.873964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:55.361013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":596,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500685+0000","last_change":"2024-09-10T20:26:21.773251+0000","last_active":"2024-09-10T20:33:29.500685+0000","last_peered":"2024-09-10T20:33:29.500685+0000","last_clean":"2024-09-10T20:33:29.500685+0000","last_became_active":"2024-09-10T20:26:19.356844+0000","last_became_peered":"2024-09-10T20:26:19.356844+0000","last_unstale":"2024-09-10T20:33:29.500685+0000","last_undegraded":"2024-09-10T20:33:29.500685+0000","last_fullsized":"2024-09-10T20:33:29.500685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:23.413224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873697+0000","last_change":"2024-09-10T20:27:17.206032+0000","last_active":"2024-09-10T20:33:29.873697+0000","last_peered":"2024-09-10T20:33:29.873697+0000","last_clean":"2024-09-10T20:33:29.873697+0000","last_became_active":"2024-09-10T20:27:17.205464+0000","last_became_peered":"2024-09-10T20:27:17.205464+0000","last_unstale":"2024-09-10T20:33:29.873697+0000","last_undegraded":"2024-09-10T20:33:29.873697+0000","last_fullsized":"2024-09-10T20:33:29.873697+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:33.219827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":593,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829238+0000","last_change":"2024-09-10T20:26:21.381255+0000","last_active":"2024-09-10T20:33:29.829238+0000","last_peered":"2024-09-10T20:33:29.829238+0000","last_clean":"2024-09-10T20:33:29.829238+0000","last_became_active":"2024-09-10T20:26:19.363220+0000","last_became_peered":"2024-09-10T20:26:19.363220+0000","last_unstale":"2024-09-10T20:33:29.829238+0000","last_undegraded":"2024-09-10T20:33:29.829238+0000","last_fullsized":"2024-09-10T20:33:29.829238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:09.279291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096679600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"23'7","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500753+0000","last_change":"2024-09-10T20:27:17.217628+0000","last_active":"2024-09-10T20:33:29.500753+0000","last_peered":"2024-09-10T20:33:29.500753+0000","last_clean":"2024-09-10T20:33:29.500753+0000","last_became_active":"2024-09-10T20:27:17.211037+0000","last_became_peered":"2024-09-10T20:27:17.211037+0000","last_unstale":"2024-09-10T20:33:29.500753+0000","last_undegraded":"2024-09-10T20:33:29.500753+0000","last_fullsized":"2024-09-10T20:33:29.500753+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:47.151425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":597,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500610+0000","last_change":"2024-09-10T20:26:21.773020+0000","last_active":"2024-09-10T20:33:29.500610+0000","last_peered":"2024-09-10T20:33:29.500610+0000","last_clean":"2024-09-10T20:33:29.500610+0000","last_became_active":"2024-09-10T20:26:19.363294+0000","last_became_peered":"2024-09-10T20:26:19.363294+0000","last_unstale":"2024-09-10T20:33:29.500610+0000","last_undegraded":"2024-09-10T20:33:29.500610+0000","last_fullsized":"2024-09-10T20:33:29.500610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:54.300619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053952999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.875890+0000","last_change":"2024-09-10T20:27:17.298070+0000","last_active":"2024-09-10T20:33:29.875890+0000","last_peered":"2024-09-10T20:33:29.875890+0000","last_clean":"2024-09-10T20:33:29.875890+0000","last_became_active":"2024-09-10T20:27:17.297745+0000","last_became_peered":"2024-09-10T20:27:17.297745+0000","last_unstale":"2024-09-10T20:33:29.875890+0000","last_undegraded":"2024-09-10T20:33:29.875890+0000","last_fullsized":"2024-09-10T20:33:29.875890+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:30:22.598793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874003+0000","last_change":"2024-09-10T20:26:21.378377+0000","last_active":"2024-09-10T20:33:29.874003+0000","last_peered":"2024-09-10T20:33:29.874003+0000","last_clean":"2024-09-10T20:33:29.874003+0000","last_became_active":"2024-09-10T20:26:19.359961+0000","last_became_peered":"2024-09-10T20:26:19.359961+0000","last_unstale":"2024-09-10T20:33:29.874003+0000","last_undegraded":"2024-09-10T20:33:29.874003+0000","last_fullsized":"2024-09-10T20:33:29.874003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:21:01.940376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829065+0000","last_change":"2024-09-10T20:27:17.282394+0000","last_active":"2024-09-10T20:33:29.829065+0000","last_peered":"2024-09-10T20:33:29.829065+0000","last_clean":"2024-09-10T20:33:29.829065+0000","last_became_active":"2024-09-10T20:27:17.281815+0000","last_became_peered":"2024-09-10T20:27:17.281815+0000","last_unstale":"2024-09-10T20:33:29.829065+0000","last_undegraded":"2024-09-10T20:33:29.829065+0000","last_fullsized":"2024-09-10T20:33:29.829065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:49.830688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":597,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876148+0000","last_change":"2024-09-10T20:26:21.386697+0000","last_active":"2024-09-10T20:33:29.876148+0000","last_peered":"2024-09-10T20:33:29.876148+0000","last_clean":"2024-09-10T20:33:29.876148+0000","last_became_active":"2024-09-10T20:26:19.360427+0000","last_became_peered":"2024-09-10T20:26:19.360427+0000","last_unstale":"2024-09-10T20:33:29.876148+0000","last_undegraded":"2024-09-10T20:33:29.876148+0000","last_fullsized":"2024-09-10T20:33:29.876148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:53.136006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020581079999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873800+0000","last_change":"2024-09-10T20:27:17.282845+0000","last_active":"2024-09-10T20:33:29.873800+0000","last_peered":"2024-09-10T20:33:29.873800+0000","last_clean":"2024-09-10T20:33:29.873800+0000","last_became_active":"2024-09-10T20:27:17.282600+0000","last_became_peered":"2024-09-10T20:27:17.282600+0000","last_unstale":"2024-09-10T20:33:29.873800+0000","last_undegraded":"2024-09-10T20:33:29.873800+0000","last_fullsized":"2024-09-10T20:33:29.873800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:38.719966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":603,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873765+0000","last_change":"2024-09-10T20:26:21.575187+0000","last_active":"2024-09-10T20:33:29.873765+0000","last_peered":"2024-09-10T20:33:29.873765+0000","last_clean":"2024-09-10T20:33:29.873765+0000","last_became_active":"2024-09-10T20:26:19.362858+0000","last_became_peered":"2024-09-10T20:26:19.362858+0000","last_unstale":"2024-09-10T20:33:29.873765+0000","last_undegraded":"2024-09-10T20:33:29.873765+0000","last_fullsized":"2024-09-10T20:33:29.873765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:22:17.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078714099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829346+0000","last_change":"2024-09-10T20:27:17.214784+0000","last_active":"2024-09-10T20:33:29.829346+0000","last_peered":"2024-09-10T20:33:29.829346+0000","last_clean":"2024-09-10T20:33:29.829346+0000","last_became_active":"2024-09-10T20:27:17.214428+0000","last_became_peered":"2024-09-10T20:27:17.214428+0000","last_unstale":"2024-09-10T20:33:29.829346+0000","last_undegraded":"2024-09-10T20:33:29.829346+0000","last_fullsized":"2024-09-10T20:33:29.829346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:50:15.983575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":610,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498452+0000","last_change":"2024-09-10T20:26:21.378915+0000","last_active":"2024-09-10T20:33:29.498452+0000","last_peered":"2024-09-10T20:33:29.498452+0000","last_clean":"2024-09-10T20:33:29.498452+0000","last_became_active":"2024-09-10T20:26:19.362158+0000","last_became_peered":"2024-09-10T20:26:19.362158+0000","last_unstale":"2024-09-10T20:33:29.498452+0000","last_undegraded":"2024-09-10T20:33:29.498452+0000","last_fullsized":"2024-09-10T20:33:29.498452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:50.708554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010122729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876226+0000","last_change":"2024-09-10T20:27:17.244860+0000","last_active":"2024-09-10T20:33:29.876226+0000","last_peered":"2024-09-10T20:33:29.876226+0000","last_clean":"2024-09-10T20:33:29.876226+0000","last_became_active":"2024-09-10T20:27:17.243935+0000","last_became_peered":"2024-09-10T20:27:17.243935+0000","last_unstale":"2024-09-10T20:33:29.876226+0000","last_undegraded":"2024-09-10T20:33:29.876226+0000","last_fullsized":"2024-09-10T20:33:29.876226+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:11.085907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":598,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500566+0000","last_change":"2024-09-10T20:26:21.773025+0000","last_active":"2024-09-10T20:33:29.500566+0000","last_peered":"2024-09-10T20:33:29.500566+0000","last_clean":"2024-09-10T20:33:29.500566+0000","last_became_active":"2024-09-10T20:26:19.359775+0000","last_became_peered":"2024-09-10T20:26:19.359775+0000","last_unstale":"2024-09-10T20:33:29.500566+0000","last_undegraded":"2024-09-10T20:33:29.500566+0000","last_fullsized":"2024-09-10T20:33:29.500566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:35.427907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054594699999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":688,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500496+0000","last_change":"2024-09-10T20:26:14.579551+0000","last_active":"2024-09-10T20:33:29.500496+0000","last_peered":"2024-09-10T20:33:29.500496+0000","last_clean":"2024-09-10T20:33:29.500496+0000","last_became_active":"2024-09-10T20:26:14.577602+0000","last_became_peered":"2024-09-10T20:26:14.577602+0000","last_unstale":"2024-09-10T20:33:29.500496+0000","last_undegraded":"2024-09-10T20:33:29.500496+0000","last_fullsized":"2024-09-10T20:33:29.500496+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_clean_scrub_stamp":"2024-09-10T20:26:13.287430+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:41.463707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.875916+0000","last_change":"2024-09-10T20:27:17.297570+0000","last_active":"2024-09-10T20:33:29.875916+0000","last_peered":"2024-09-10T20:33:29.875916+0000","last_clean":"2024-09-10T20:33:29.875916+0000","last_became_active":"2024-09-10T20:27:17.295709+0000","last_became_peered":"2024-09-10T20:27:17.295709+0000","last_unstale":"2024-09-10T20:33:29.875916+0000","last_undegraded":"2024-09-10T20:33:29.875916+0000","last_fullsized":"2024-09-10T20:33:29.875916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:37.519789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498888+0000","last_change":"2024-09-10T20:27:17.219710+0000","last_active":"2024-09-10T20:33:29.498888+0000","last_peered":"2024-09-10T20:33:29.498888+0000","last_clean":"2024-09-10T20:33:29.498888+0000","last_became_active":"2024-09-10T20:27:17.217973+0000","last_became_peered":"2024-09-10T20:27:17.217973+0000","last_unstale":"2024-09-10T20:33:29.498888+0000","last_undegraded":"2024-09-10T20:33:29.498888+0000","last_fullsized":"2024-09-10T20:33:29.498888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:55.139486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500679+0000","last_change":"2024-09-10T20:27:17.318458+0000","last_active":"2024-09-10T20:33:29.500679+0000","last_peered":"2024-09-10T20:33:29.500679+0000","last_clean":"2024-09-10T20:33:29.500679+0000","last_became_active":"2024-09-10T20:27:17.318202+0000","last_became_peered":"2024-09-10T20:27:17.318202+0000","last_unstale":"2024-09-10T20:33:29.500679+0000","last_undegraded":"2024-09-10T20:33:29.500679+0000","last_fullsized":"2024-09-10T20:33:29.500679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:49.111756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873865+0000","last_change":"2024-09-10T20:27:17.240732+0000","last_active":"2024-09-10T20:33:29.873865+0000","last_peered":"2024-09-10T20:33:29.873865+0000","last_clean":"2024-09-10T20:33:29.873865+0000","last_became_active":"2024-09-10T20:27:17.240435+0000","last_became_peered":"2024-09-10T20:27:17.240435+0000","last_unstale":"2024-09-10T20:33:29.873865+0000","last_undegraded":"2024-09-10T20:33:29.873865+0000","last_fullsized":"2024-09-10T20:33:29.873865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:54:42.900319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874276+0000","last_change":"2024-09-10T20:27:17.230157+0000","last_active":"2024-09-10T20:33:29.874276+0000","last_peered":"2024-09-10T20:33:29.874276+0000","last_clean":"2024-09-10T20:33:29.874276+0000","last_became_active":"2024-09-10T20:27:17.229886+0000","last_became_peered":"2024-09-10T20:27:17.229886+0000","last_unstale":"2024-09-10T20:33:29.874276+0000","last_undegraded":"2024-09-10T20:33:29.874276+0000","last_fullsized":"2024-09-10T20:33:29.874276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:40.265668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874302+0000","last_change":"2024-09-10T20:27:17.214355+0000","last_active":"2024-09-10T20:33:29.874302+0000","last_peered":"2024-09-10T20:33:29.874302+0000","last_clean":"2024-09-10T20:33:29.874302+0000","last_became_active":"2024-09-10T20:27:17.213875+0000","last_became_peered":"2024-09-10T20:27:17.213875+0000","last_unstale":"2024-09-10T20:33:29.874302+0000","last_undegraded":"2024-09-10T20:33:29.874302+0000","last_fullsized":"2024-09-10T20:33:29.874302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:55.411553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873798+0000","last_change":"2024-09-10T20:27:17.288406+0000","last_active":"2024-09-10T20:33:29.873798+0000","last_peered":"2024-09-10T20:33:29.873798+0000","last_clean":"2024-09-10T20:33:29.873798+0000","last_became_active":"2024-09-10T20:27:17.288092+0000","last_became_peered":"2024-09-10T20:27:17.288092+0000","last_unstale":"2024-09-10T20:33:29.873798+0000","last_undegraded":"2024-09-10T20:33:29.873798+0000","last_fullsized":"2024-09-10T20:33:29.873798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:47.584995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873374+0000","last_change":"2024-09-10T20:27:17.210249+0000","last_active":"2024-09-10T20:33:29.873374+0000","last_peered":"2024-09-10T20:33:29.873374+0000","last_clean":"2024-09-10T20:33:29.873374+0000","last_became_active":"2024-09-10T20:27:17.209363+0000","last_became_peered":"2024-09-10T20:27:17.209363+0000","last_unstale":"2024-09-10T20:33:29.873374+0000","last_undegraded":"2024-09-10T20:33:29.873374+0000","last_fullsized":"2024-09-10T20:33:29.873374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:19.942713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500468+0000","last_change":"2024-09-10T20:27:17.225434+0000","last_active":"2024-09-10T20:33:29.500468+0000","last_peered":"2024-09-10T20:33:29.500468+0000","last_clean":"2024-09-10T20:33:29.500468+0000","last_became_active":"2024-09-10T20:27:17.224230+0000","last_became_peered":"2024-09-10T20:27:17.224230+0000","last_unstale":"2024-09-10T20:33:29.500468+0000","last_undegraded":"2024-09-10T20:33:29.500468+0000","last_fullsized":"2024-09-10T20:33:29.500468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:49:09.800954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500670+0000","last_change":"2024-09-10T20:27:17.217944+0000","last_active":"2024-09-10T20:33:29.500670+0000","last_peered":"2024-09-10T20:33:29.500670+0000","last_clean":"2024-09-10T20:33:29.500670+0000","last_became_active":"2024-09-10T20:27:17.217107+0000","last_became_peered":"2024-09-10T20:27:17.217107+0000","last_unstale":"2024-09-10T20:33:29.500670+0000","last_undegraded":"2024-09-10T20:33:29.500670+0000","last_fullsized":"2024-09-10T20:33:29.500670+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:52.708045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829272+0000","last_change":"2024-09-10T20:27:17.283023+0000","last_active":"2024-09-10T20:33:29.829272+0000","last_peered":"2024-09-10T20:33:29.829272+0000","last_clean":"2024-09-10T20:33:29.829272+0000","last_became_active":"2024-09-10T20:27:17.282074+0000","last_became_peered":"2024-09-10T20:27:17.282074+0000","last_unstale":"2024-09-10T20:33:29.829272+0000","last_undegraded":"2024-09-10T20:33:29.829272+0000","last_fullsized":"2024-09-10T20:33:29.829272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:51.986119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874322+0000","last_change":"2024-09-10T20:27:17.211220+0000","last_active":"2024-09-10T20:33:29.874322+0000","last_peered":"2024-09-10T20:33:29.874322+0000","last_clean":"2024-09-10T20:33:29.874322+0000","last_became_active":"2024-09-10T20:27:17.210803+0000","last_became_peered":"2024-09-10T20:27:17.210803+0000","last_unstale":"2024-09-10T20:33:29.874322+0000","last_undegraded":"2024-09-10T20:33:29.874322+0000","last_fullsized":"2024-09-10T20:33:29.874322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:38.315319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876046+0000","last_change":"2024-09-10T20:27:17.311360+0000","last_active":"2024-09-10T20:33:29.876046+0000","last_peered":"2024-09-10T20:33:29.876046+0000","last_clean":"2024-09-10T20:33:29.876046+0000","last_became_active":"2024-09-10T20:27:17.310182+0000","last_became_peered":"2024-09-10T20:27:17.310182+0000","last_unstale":"2024-09-10T20:33:29.876046+0000","last_undegraded":"2024-09-10T20:33:29.876046+0000","last_fullsized":"2024-09-10T20:33:29.876046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:34:02.949919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574956,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80416,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291424,"statfs":{"total":96636764160,"available":96554418176,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574955,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80488,"kb_used_data":4872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291352,"statfs":{"total":96636764160,"available":96554344448,"internally_reserved":0,"allocated":4988928,"data_stored":2922726,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574953,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76260,"kb_used_data":4812,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295580,"statfs":{"total":96636764160,"available":96558673920,"internally_reserved":0,"allocated":4927488,"data_stored":2393812,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640363,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72172,"kb_used_data":4868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299668,"statfs":{"total":96636764160,"available":96562860032,"internally_reserved":0,"allocated":4984832,"data_stored":2922721,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607659,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67944,"kb_used_data":4816,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303896,"statfs":{"total":96636764160,"available":96567189504,"internally_reserved":0,"allocated":4931584,"data_stored":2393817,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607658,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72096,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299744,"statfs":{"total":96636764160,"available":96562937856,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:34:52.880 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-10T20:34:53.003 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.997+0000 7f28d3fde640 10 monclient: get_monmap_and_config 2024-09-10T20:34:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.997+0000 7f28d3fde640 10 monclient: build_initial_monmap 2024-09-10T20:34:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.997+0000 7f28d3fde640 10 monclient: monmap: 2024-09-10T20:34:53.004 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:53.005 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:53.005 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:53.000906+0000 2024-09-10T20:34:53.005 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:53.000906+0000 2024-09-10T20:34:53.005 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:53.005 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.997+0000 7f28d3fde640 1 Processor -- start 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.997+0000 7f28d3fde640 1 -- start start 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:52.997+0000 7f28d3fde640 10 monclient: init 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 10 monclient: _add_conns ranks=[2,4,0,3,1] 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a41a0 0x7f28cc0a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 10 monclient(hunting): picked mon.noname-b con 0x7f28cc0a41a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:53.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f28cc0a4580 0x7f28cc0a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 10 monclient(hunting): picked mon.noname-c con 0x7f28cc0a4580 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f28cc0a49a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28cc0a6a20 con 0x7f28cc0a49a0 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f28cc0a2570 con 0x7f28cc0a4580 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f28cc070400 con 0x7f28cc0a41a0 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d3fde640 10 monclient(hunting): authenticate will time out at 1453.140548s 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d257b640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f28cc0a49a0 0x7f28cc0a5f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60720/0 (socket says 172.21.15.153:60720) 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d257b640 1 -- 172.21.15.153:0/627926500 learned_addr learned my addr 172.21.15.153:0/627926500 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 1 --2- 172.21.15.153:0/627926500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a41a0 0x7f28cc0a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:53.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient(hunting): get_auth_request con 0x7f28cc0a41a0 auth_method 0 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1579640 1 --2- 172.21.15.153:0/627926500 >> v2:172.21.15.176:3300/0 conn(0x7f28cc0a4580 0x7f28cc0a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient(hunting): handle_auth_done global_id 54131 payload 274 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 1 -- 172.21.15.153:0/627926500 >> v2:172.21.15.176:3300/0 conn(0x7f28cc0a4580 msgr2=0x7f28cc0a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:53.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 1 --2- 172.21.15.153:0/627926500 >> v2:172.21.15.176:3300/0 conn(0x7f28cc0a4580 0x7f28cc0a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 1 -- 172.21.15.153:0/627926500 >> v1:172.21.15.153:6789/0 conn(0x7f28cc0a49a0 legacy=0x7f28cc0a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient: _finish_hunting 0 2024-09-10T20:34:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 1 monclient: found mon.noname-b 2024-09-10T20:34:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1d7a640 1 -- 172.21.15.153:0/627926500 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28cc0a3c80 con 0x7f28cc0a41a0 2024-09-10T20:34:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1579640 10 monclient: get_auth_request con 0x7f28cc0a4580 auth_method 0 2024-09-10T20:34:53.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.001+0000 7f28d1579640 1 --2- 172.21.15.153:0/627926500 >> v2:172.21.15.176:3300/0 conn(0x7f28cc0a4580 0x7f28cc0a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d1d7a640 1 -- 172.21.15.153:0/627926500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a41a0 msgr2=0x7f28cc0a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d1d7a640 1 --2- 172.21.15.153:0/627926500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a41a0 0x7f28cc0a3740 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f28bc0097c0 tx=0x7f28bc01c360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f01662bedb04109e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 1 -- 172.21.15.153:0/627926500 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc0b0070 con 0x7f28cc0a41a0 2024-09-10T20:34:53.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 10 monclient: dump: 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:53.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 10 monclient: _finish_auth 0 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 10 monclient: _check_auth_tickets 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 1 -- 172.21.15.153:0/627926500 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28bc0ab600 con 0x7f28cc0a41a0 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 1 -- 172.21.15.153:0/627926500 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc0ab9d0 con 0x7f28cc0a41a0 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d0d78640 10 monclient: dump: 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:53.017 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 5 monclient: authenticate success, global_id 54131 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 4 set_mon_vals no callback set 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 10 monclient: shutdown 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 -- 172.21.15.153:0/627926500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a41a0 msgr2=0x7f28cc0a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:53.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 --2- 172.21.15.153:0/627926500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a41a0 0x7f28cc0a3740 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f28bc0097c0 tx=0x7f28bc01c360 comp rx=0 tx=0).stop 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 -- 172.21.15.153:0/627926500 shutdown_connections 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 --2- 172.21.15.153:0/627926500 >> v2:172.21.15.176:3300/0 conn(0x7f28cc0a4580 0x7f28cc0a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 --2- 172.21.15.153:0/627926500 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a41a0 0x7f28cc0a3740 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 -- 172.21.15.153:0/627926500 >> 172.21.15.153:0/627926500 conn(0x7f28cc0ab7e0 msgr2=0x7f28cc0a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 -- 172.21.15.153:0/627926500 shutdown_connections 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 -- 172.21.15.153:0/627926500 wait complete. 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 10 monclient: build_initial_monmap 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 10 monclient: monmap: 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:53.019 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 librados: starting msgr at 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 librados: starting objecter 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 Processor -- start 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 -- start start 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 librados: setting wanted keys 2024-09-10T20:34:53.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 1 librados: calling monclient init 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.005+0000 7f28d3fde640 10 monclient: init 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28cc0a41a0 0x7f28cc128930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient(hunting): picked mon.noname-b con 0x7f28cc0a41a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28cc0a4580 0x7f28cc13f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient(hunting): picked mon.noname-a con 0x7f28cc0a4580 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a49a0 0x7f28cc1263f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient(hunting): picked mon.noname-c con 0x7f28cc0a49a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:53.021 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc143210 con 0x7f28cc0a4580 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f28cc0a6830 con 0x7f28cc0a41a0 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f28cc143390 con 0x7f28cc0a49a0 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient(hunting): authenticate will time out at 1453.148235s 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a49a0 0x7f28cc1263f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1579640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28cc0a4580 0x7f28cc13f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a49a0 0x7f28cc1263f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37392/0 (socket says 172.21.15.153:37392) 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 -- 172.21.15.153:0/3798522810 learned_addr learned my addr 172.21.15.153:0/3798522810 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient(hunting): get_auth_request con 0x7f28cc0a49a0 auth_method 0 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:53.022 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1579640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28cc0a4580 0x7f28cc13f7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56192/0 (socket says 172.21.15.153:56192) 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1579640 10 monclient(hunting): get_auth_request con 0x7f28cc0a4580 auth_method 0 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1579640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1579640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1579640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1579640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1579640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1579640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1d7a640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28cc0a41a0 0x7f28cc128930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient(hunting): handle_auth_done global_id 54134 payload 824 2024-09-10T20:34:53.023 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 -- 172.21.15.153:0/3798522810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28cc0a41a0 msgr2=0x7f28cc128930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28cc0a41a0 0x7f28cc128930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 -- 172.21.15.153:0/3798522810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28cc0a4580 msgr2=0x7f28cc13f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28cc0a4580 0x7f28cc13f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient: _finish_hunting 0 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 monclient: found mon.noname-c 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 -- 172.21.15.153:0/3798522810 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28cc0a21c0 con 0x7f28cc0a49a0 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1d7a640 10 monclient: get_auth_request con 0x7f28cc0a41a0 auth_method 0 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d1d7a640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28cc0a41a0 0x7f28cc128930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 -- 172.21.15.153:0/3798522810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a49a0 msgr2=0x7f28cc1263f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d257b640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a49a0 0x7f28cc1263f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f28c400caf0 tx=0x7f28c400a060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:53.024 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 client.?.objecter ms_handle_connect 0x7f28cc0a49a0 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 1 -- 172.21.15.153:0/3798522810 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4012400 con 0x7f28cc0a49a0 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 monclient: dump: 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:53.025 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 monclient: _finish_auth 0 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 monclient: _check_auth_tickets 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 1 -- 172.21.15.153:0/3798522810 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c40125c0 con 0x7f28cc0a49a0 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 5 monclient: authenticate success, global_id 54134 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient: _renew_subs 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28cc129e50 con 0x7f28cc0a49a0 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 1 -- 172.21.15.153:0/3798522810 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c40129b0 con 0x7f28cc0a49a0 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c27fc640 10 monclient: dump: 2024-09-10T20:34:53.026 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28c3fff640 4 set_mon_vals no callback set 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 client.54134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient: _renew_subs 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f28cc127dd0 con 0x7f28cc0a49a0 2024-09-10T20:34:53.027 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 librados: init done 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.009+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28cc0a3740 con 0x7f28cc0a49a0 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28c27fc640 1 -- 172.21.15.153:0/3798522810 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f28c4012bd0 con 0x7f28cc0a49a0 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28c27fc640 1 --2- 172.21.15.153:0/3798522810 >> v2:172.21.15.176:6812/3003473398 conn(0x7f28b003ced0 0x7f28b003f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28c27fc640 1 -- 172.21.15.153:0/3798522810 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f28c404fd20 con 0x7f28cc0a49a0 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28c27fc640 10 client.54134.objecter ms_dispatch 0x7f28cc000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28c27fc640 3 client.54134.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28c27fc640 3 client.54134.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28c27fc640 20 client.54134.objecter dump_active .. 0 homeless 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28d1d7a640 1 --2- 172.21.15.153:0/3798522810 >> v2:172.21.15.176:6812/3003473398 conn(0x7f28b003ced0 0x7f28b003f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:53.028 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28d1d7a640 10 monclient: get_auth_request con 0x7f28b003ced0 auth_method 0 2024-09-10T20:34:53.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28c27fc640 10 client.54134.objecter ms_handle_connect 0x7f28b003ced0 2024-09-10T20:34:53.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.013+0000 7f28d1d7a640 1 --2- 172.21.15.153:0/3798522810 >> v2:172.21.15.176:6812/3003473398 conn(0x7f28b003ced0 0x7f28b003f390 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0a7aa0 tx=0x7f28bc008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:53.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.017+0000 7f28c27fc640 1 -- 172.21.15.153:0/3798522810 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f28cc0a21c0 con 0x7f28cc0a49a0 2024-09-10T20:34:53.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.017+0000 7f28c27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:53.029 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.017+0000 7f28c27fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:53.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.185+0000 7f28d3fde640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:34:53.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.185+0000 7f28d3fde640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:34:53.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.185+0000 7f28d3fde640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:53.189 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.185+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f28cc06cd40 con 0x7f28cc0a49a0 2024-09-10T20:34:53.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28c27fc640 1 -- 172.21.15.153:0/3798522810 <== mon.2 v2:172.21.15.153:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v317) ==== 74+0+13528 (secure 0 0 0) 0x7f28c4010d10 con 0x7f28cc0a49a0 2024-09-10T20:34:53.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28c27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-10T20:34:53.191 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28c27fc640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:34:53.191 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:34:53.191 INFO:teuthology.orchestra.run.smithi153.stdout:{"epoch":317,"fsid":"8126fdd8-2c77-4da0-ba28-d5fdc347eec8","created":"2024-09-10T20:25:59.587062+0000","modified":"2024-09-10T20:33:29.489593+0000","last_up_change":"2024-09-10T20:26:12.282988+0000","last_in_change":"2024-09-10T20:26:02.414479+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":102,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":259,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-10T20:26:13.125072+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-10T20:26:18.322750+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-10T20:27:16.133152+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.309999942779541,"score_stable":1.309999942779541,"optimal_score":1,"raw_score_acting":1.309999942779541,"raw_score_stable":1.309999942779541,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"faf817a9-b230-4970-b8c5-e5637fd29193","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":306,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6800","nonce":4216589932}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6801","nonce":4216589932}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6803","nonce":4216589932}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6802","nonce":4216589932}]},"public_addr":"172.21.15.153:6800/4216589932","cluster_addr":"172.21.15.153:6801/4216589932","heartbeat_back_addr":"172.21.15.153:6803/4216589932","heartbeat_front_addr":"172.21.15.153:6802/4216589932","state":["exists","up"]},{"osd":1,"uuid":"f85962f8-8bd9-4e26-8d66-49b6528617a3","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6804","nonce":1715391555}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6805","nonce":1715391555}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6807","nonce":1715391555}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6806","nonce":1715391555}]},"public_addr":"172.21.15.153:6804/1715391555","cluster_addr":"172.21.15.153:6805/1715391555","heartbeat_back_addr":"172.21.15.153:6807/1715391555","heartbeat_front_addr":"172.21.15.153:6806/1715391555","state":["exists","up"]},{"osd":2,"uuid":"18d54b55-1393-42b9-9fcc-84202659cf31","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6808","nonce":1626005750}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6809","nonce":1626005750}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6811","nonce":1626005750}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.153:6810","nonce":1626005750}]},"public_addr":"172.21.15.153:6808/1626005750","cluster_addr":"172.21.15.153:6809/1626005750","heartbeat_back_addr":"172.21.15.153:6811/1626005750","heartbeat_front_addr":"172.21.15.153:6810/1626005750","state":["exists","up"]},{"osd":3,"uuid":"5149b9c4-35d7-4ffb-9fac-7b3d9b660101","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":306,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6808","nonce":2953083416}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6809","nonce":2953083416}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6811","nonce":2953083416}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6810","nonce":2953083416}]},"public_addr":"172.21.15.176:6808/2953083416","cluster_addr":"172.21.15.176:6809/2953083416","heartbeat_back_addr":"172.21.15.176:6811/2953083416","heartbeat_front_addr":"172.21.15.176:6810/2953083416","state":["exists","up"]},{"osd":4,"uuid":"61e0c0cc-4301-4947-9f1d-838d2d7bbfea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6804","nonce":4030285519}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6805","nonce":4030285519}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6807","nonce":4030285519}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6806","nonce":4030285519}]},"public_addr":"172.21.15.176:6804/4030285519","cluster_addr":"172.21.15.176:6805/4030285519","heartbeat_back_addr":"172.21.15.176:6807/4030285519","heartbeat_front_addr":"172.21.15.176:6806/4030285519","state":["exists","up"]},{"osd":5,"uuid":"2feacd82-c45a-4d90-8210-ff9faecc2d79","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":313,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6800","nonce":2372911183}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6801","nonce":2372911183}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6803","nonce":2372911183}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.176:6802","nonce":2372911183}]},"public_addr":"172.21.15.176:6800/2372911183","cluster_addr":"172.21.15.176:6801/2372911183","heartbeat_back_addr":"172.21.15.176:6803/2372911183","heartbeat_front_addr":"172.21.15.176:6802/2372911183","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.301670+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364442+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:10.037323+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.733818+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.364748+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-10T20:26:09.173071+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":11,"snaps":[{"begin":1,"length":4}]},{"pool":12,"snaps":[{"begin":2,"length":5}]},{"pool":25,"snaps":[{"begin":1,"length":1}]},{"pool":61,"snaps":[{"begin":1,"length":1}]},{"pool":90,"snaps":[{"begin":4,"length":1},{"begin":7,"length":2}]},{"pool":100,"snaps":[{"begin":1,"length":1}]},{"pool":117,"snaps":[{"begin":1,"length":2}]},{"pool":131,"snaps":[{"begin":3,"length":1}]},{"pool":138,"snaps":[{"begin":1,"length":4}]},{"pool":145,"snaps":[{"begin":2,"length":2}]},{"pool":155,"snaps":[{"begin":2,"length":3}]},{"pool":162,"snaps":[{"begin":2,"length":4}]},{"pool":165,"snaps":[{"begin":2,"length":2}]},{"pool":174,"snaps":[{"begin":2,"length":1}]},{"pool":179,"snaps":[{"begin":1,"length":4}]},{"pool":190,"snaps":[{"begin":4,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-10T20:34:53.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 10 librados: watch_flush enter 2024-09-10T20:34:53.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 10 librados: watch_flush exit 2024-09-10T20:34:53.193 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 20 client.54134.objecter shutdown clearing up homeless session... 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 10 client.54134.objecter successfully canceled tick 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 >> v2:172.21.15.176:6812/3003473398 conn(0x7f28b003ced0 msgr2=0x7f28b003f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 --2- 172.21.15.153:0/3798522810 >> v2:172.21.15.176:6812/3003473398 conn(0x7f28b003ced0 0x7f28b003f390 secure :-1 s=READY pgs=3375 cs=0 l=1 rev1=1 crypto rx=0x7f28cc0a7aa0 tx=0x7f28bc008040 comp rx=0 tx=0).stop 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 10 monclient: shutdown 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a49a0 msgr2=0x7f28cc1263f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a49a0 0x7f28cc1263f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f28c400caf0 tx=0x7f28c400a060 comp rx=0 tx=0).stop 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 shutdown_connections 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 --2- 172.21.15.153:0/3798522810 >> v2:172.21.15.176:6812/3003473398 conn(0x7f28b003ced0 0x7f28b003f390 unknown :-1 s=CLOSED pgs=3375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f28cc0a49a0 0x7f28cc1263f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f28cc0a4580 0x7f28cc13f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:53.194 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 --2- 172.21.15.153:0/3798522810 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f28cc0a41a0 0x7f28cc128930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:53.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 >> 172.21.15.153:0/3798522810 conn(0x7f28cc0ab7e0 msgr2=0x7f28cc0a0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:53.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 shutdown_connections 2024-09-10T20:34:53.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 -- 172.21.15.153:0/3798522810 wait complete. 2024-09-10T20:34:53.195 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:53.189+0000 7f28d3fde640 1 librados: shutdown 2024-09-10T20:34:54.211 INFO:tasks.ceph:Scrubbing osd.0 2024-09-10T20:34:54.212 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:34:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient: get_monmap_and_config 2024-09-10T20:34:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient: build_initial_monmap 2024-09-10T20:34:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient: monmap: 2024-09-10T20:34:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.347 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:54.347124+0000 2024-09-10T20:34:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:54.347124+0000 2024-09-10T20:34:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:54.348 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:54.348 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 1 Processor -- start 2024-09-10T20:34:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 1 -- start start 2024-09-10T20:34:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient: init 2024-09-10T20:34:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-10T20:34:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f08dc09e2b0 0x7f08dc09e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.350 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f08dc09e2b0 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f08dc09ebd0 0x7f08dc135510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f08dc09ebd0 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 0x7f08dc135ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f08dc135a50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f08dc0a34f0 con 0x7f08dc09e2b0 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f08dc0771a0 con 0x7f08dc09ebd0 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f08dc0a2b00 con 0x7f08dc135a50 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08e1a0e640 10 monclient(hunting): authenticate will time out at 1454.486902s 2024-09-10T20:34:54.351 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f08dc09e2b0 0x7f08dc09e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f08dc09e2b0 0x7f08dc09e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56200/0 (socket says 172.21.15.153:56200) 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 1 -- 172.21.15.153:0/332438521 learned_addr learned my addr 172.21.15.153:0/332438521 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient(hunting): get_auth_request con 0x7f08dc09e2b0 auth_method 0 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08db7fe640 1 --2- 172.21.15.153:0/332438521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 0x7f08dc135ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08db7fe640 10 monclient(hunting): get_auth_request con 0x7f08dc135a50 auth_method 0 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient(hunting): handle_auth_done global_id 34398 payload 274 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08da7fc640 1 --2- 172.21.15.153:0/332438521 >> v2:172.21.15.176:3300/0 conn(0x7f08dc09ebd0 0x7f08dc135510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 1 -- 172.21.15.153:0/332438521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 msgr2=0x7f08dc135ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 1 --2- 172.21.15.153:0/332438521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 0x7f08dc135ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 1 -- 172.21.15.153:0/332438521 >> v2:172.21.15.176:3300/0 conn(0x7f08dc09ebd0 msgr2=0x7f08dc135510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 1 --2- 172.21.15.153:0/332438521 >> v2:172.21.15.176:3300/0 conn(0x7f08dc09ebd0 0x7f08dc135510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient: _finish_hunting 0 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 1 monclient: found mon.noname-a 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.345+0000 7f08daffd640 1 -- 172.21.15.153:0/332438521 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08dc0a2d80 con 0x7f08dc09e2b0 2024-09-10T20:34:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08daffd640 1 -- 172.21.15.153:0/332438521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09e2b0 msgr2=0x7f08dc09e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08daffd640 1 --2- 172.21.15.153:0/332438521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09e2b0 0x7f08dc09e690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f08dc09f630 tx=0x7f08d401c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3c8fc74b14b4c5e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 1 -- 172.21.15.153:0/332438521 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d40a55c0 con 0x7f08dc09e2b0 2024-09-10T20:34:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 10 monclient: dump: 2024-09-10T20:34:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.357 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 10 monclient: _finish_auth 0 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:34:54.358 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 1 -- 172.21.15.153:0/332438521 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08d40a5760 con 0x7f08dc09e2b0 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08e1a0e640 5 monclient: authenticate success, global_id 34398 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 1 -- 172.21.15.153:0/332438521 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d40a5b30 con 0x7f08dc09e2b0 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08d9ffb640 10 monclient: dump: 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.359 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08e1a0e640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.349+0000 7f08e1a0e640 4 set_mon_vals no callback set 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient: shutdown 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 -- 172.21.15.153:0/332438521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09e2b0 msgr2=0x7f08dc09e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 --2- 172.21.15.153:0/332438521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09e2b0 0x7f08dc09e690 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f08dc09f630 tx=0x7f08d401c940 comp rx=0 tx=0).stop 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 -- 172.21.15.153:0/332438521 shutdown_connections 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 --2- 172.21.15.153:0/332438521 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 0x7f08dc135ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 --2- 172.21.15.153:0/332438521 >> v2:172.21.15.176:3300/0 conn(0x7f08dc09ebd0 0x7f08dc135510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 --2- 172.21.15.153:0/332438521 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09e2b0 0x7f08dc09e690 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.360 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 -- 172.21.15.153:0/332438521 >> 172.21.15.153:0/332438521 conn(0x7f08dc0a7900 msgr2=0x7f08dc0a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 -- 172.21.15.153:0/332438521 shutdown_connections 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 -- 172.21.15.153:0/332438521 wait complete. 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient: build_initial_monmap 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient: monmap: 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.361 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 librados: starting msgr at 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 librados: starting objecter 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 Processor -- start 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 -- start start 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 librados: setting wanted keys 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 librados: calling monclient init 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient: init 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08dc09e2b0 0x7f08dc130d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f08dc09e2b0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09ebd0 0x7f08dc06f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f08dc09ebd0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 0x7f08dc06f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f08dc135a50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:54.362 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc0a4330 con 0x7f08dc09ebd0 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc0a41b0 con 0x7f08dc09e2b0 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f08dc09efe0 con 0x7f08dc135a50 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08e1a0e640 10 monclient(hunting): authenticate will time out at 1454.494209s 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09ebd0 0x7f08dc06f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08db7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 0x7f08dc06f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09ebd0 0x7f08dc06f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56212/0 (socket says 172.21.15.153:56212) 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 1 -- 172.21.15.153:0/2607457677 learned_addr learned my addr 172.21.15.153:0/2607457677 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:54.363 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient(hunting): get_auth_request con 0x7f08dc09ebd0 auth_method 0 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08db7fe640 10 monclient(hunting): get_auth_request con 0x7f08dc135a50 auth_method 0 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08db7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08db7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08db7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08db7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08db7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.364 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient(hunting): handle_auth_done global_id 34401 payload 824 2024-09-10T20:34:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08daffd640 1 --2- 172.21.15.153:0/2607457677 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08dc09e2b0 0x7f08dc130d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 1 -- 172.21.15.153:0/2607457677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 msgr2=0x7f08dc06f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 1 --2- 172.21.15.153:0/2607457677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 0x7f08dc06f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 1 -- 172.21.15.153:0/2607457677 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08dc09e2b0 msgr2=0x7f08dc130d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 1 --2- 172.21.15.153:0/2607457677 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08dc09e2b0 0x7f08dc130d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.365 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient: _finish_hunting 0 2024-09-10T20:34:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 1 monclient: found mon.noname-a 2024-09-10T20:34:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08da7fc640 1 -- 172.21.15.153:0/2607457677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08dc138ec0 con 0x7f08dc09ebd0 2024-09-10T20:34:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08daffd640 10 monclient: get_auth_request con 0x7f08dc09e2b0 auth_method 0 2024-09-10T20:34:54.366 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08db7fe640 1 --2- 172.21.15.153:0/2607457677 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f08dc135a50 0x7f08dc06f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:54.367 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.353+0000 7f08daffd640 1 --2- 172.21.15.153:0/2607457677 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f08dc09e2b0 0x7f08dc130d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:54.369 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08da7fc640 1 -- 172.21.15.153:0/2607457677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09ebd0 msgr2=0x7f08dc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08da7fc640 1 --2- 172.21.15.153:0/2607457677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09ebd0 0x7f08dc06f0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f08c400c7f0 tx=0x7f08c400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 client.?.objecter ms_handle_connect 0x7f08dc09ebd0 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 1 -- 172.21.15.153:0/2607457677 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4015070 con 0x7f08dc09ebd0 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: dump: 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.370 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: _finish_auth 0 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: _check_auth_tickets 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 1 -- 172.21.15.153:0/2607457677 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08c4004240 con 0x7f08dc09ebd0 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 5 monclient: authenticate success, global_id 34401 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 10 monclient: _renew_subs 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08dc1391a0 con 0x7f08dc09ebd0 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:54.371 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 1 -- 172.21.15.153:0/2607457677 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c4007830 con 0x7f08dc09ebd0 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d97fa640 4 set_mon_vals no callback set 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: dump: 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.372 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 10 client.34401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 10 monclient: _renew_subs 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08dc0a59d0 con 0x7f08dc09ebd0 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 1 librados: init done 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 10 client.34401.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 15 client.34401.objecter _session_command_op_assign -1 1 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 20 client.34401.objecter _get_session osd=-1 returning homeless 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 10 client.34401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f08dc137a30 con 0x7f08dc09ebd0 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 1 -- 172.21.15.153:0/2607457677 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f08c4007a10 con 0x7f08dc09ebd0 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 1 --2- 172.21.15.153:0/2607457677 >> v2:172.21.15.176:6812/3003473398 conn(0x7f08bc03cea0 0x7f08bc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.373 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 1 -- 172.21.15.153:0/2607457677 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f08c404f530 con 0x7f08dc09ebd0 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 client.34401.objecter ms_dispatch 0x7f08dc000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 3 client.34401.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 3 client.34401.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 client.34401.objecter checking command 1 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 1 --2- 172.21.15.153:0/2607457677 >> v2:172.21.15.153:6800/4216589932 conn(0x7f08bc043d70 0x7f08bc046210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 20 client.34401.objecter _get_session s=0x7f08bc0436c0 osd=0 3 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 20 client.34401.objecter put_session s=0x7f08bc0436c0 osd=0 3 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 15 client.34401.objecter _session_command_op_remove -1 1 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 20 client.34401.objecter _get_session s=0x7f08bc0436c0 osd=0 3 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 20 client.34401.objecter get_session s=0x7f08bc0436c0 osd=0 3 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 15 client.34401.objecter _session_command_op_assign 0 1 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 20 client.34401.objecter put_session s=0x7f08bc0436c0 osd=0 4 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 client.34401.objecter _send_command 1 2024-09-10T20:34:54.374 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 1 -- 172.21.15.153:0/2607457677 --> v2:172.21.15.153:6800/4216589932 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f08bc0468a0 con 0x7f08bc043d70 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 20 client.34401.objecter dump_active .. 0 homeless 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 1 -- 172.21.15.153:0/2607457677 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=317) ==== 24+0+0 (secure 0 0 0) 0x7f08c4056050 con 0x7f08dc09ebd0 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08d3fff640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08db7fe640 1 --2- 172.21.15.153:0/2607457677 >> v2:172.21.15.153:6800/4216589932 conn(0x7f08bc043d70 0x7f08bc046210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08db7fe640 10 monclient: get_auth_request con 0x7f08bc043d70 auth_method 0 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08daffd640 1 --2- 172.21.15.153:0/2607457677 >> v2:172.21.15.176:6812/3003473398 conn(0x7f08bc03cea0 0x7f08bc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.357+0000 7f08daffd640 10 monclient: get_auth_request con 0x7f08bc03cea0 auth_method 0 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.361+0000 7f08d3fff640 10 client.34401.objecter ms_handle_connect 0x7f08bc03cea0 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.361+0000 7f08daffd640 1 --2- 172.21.15.153:0/2607457677 >> v2:172.21.15.176:6812/3003473398 conn(0x7f08bc03cea0 0x7f08bc03f360 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f08d40ab000 tx=0x7f08d40a7070 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.361+0000 7f08d3fff640 10 client.34401.objecter ms_handle_connect 0x7f08bc043d70 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.361+0000 7f08db7fe640 1 --2- 172.21.15.153:0/2607457677 >> v2:172.21.15.153:6800/4216589932 conn(0x7f08bc043d70 0x7f08bc046210 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.365+0000 7f08d3fff640 1 -- 172.21.15.153:0/2607457677 <== osd.0 v2:172.21.15.153:6800/4216589932 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f08bc0468a0 con 0x7f08bc043d70 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.365+0000 7f08d3fff640 10 client.34401.objecter ms_dispatch 0x7f08dc000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:34:54.375 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.365+0000 7f08d3fff640 10 client.34401.objecter _finish_command 1 = system:0 2024-09-10T20:34:54.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.365+0000 7f08d3fff640 20 client.34401.objecter put_session s=0x7f08bc0436c0 osd=0 4 2024-09-10T20:34:54.376 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.365+0000 7f08d3fff640 15 client.34401.objecter _session_command_op_remove 0 1 2024-09-10T20:34:54.379 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 10 client.34401.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-10T20:34:54.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 15 client.34401.objecter _session_command_op_assign -1 2 2024-09-10T20:34:54.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 20 client.34401.objecter _get_session s=0x7f08bc0436c0 osd=0 3 2024-09-10T20:34:54.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 20 client.34401.objecter put_session s=0x7f08bc0436c0 osd=0 3 2024-09-10T20:34:54.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 20 client.34401.objecter _get_session s=0x7f08bc0436c0 osd=0 3 2024-09-10T20:34:54.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 15 client.34401.objecter _session_command_op_remove -1 2 2024-09-10T20:34:54.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 20 client.34401.objecter get_session s=0x7f08bc0436c0 osd=0 3 2024-09-10T20:34:54.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 15 client.34401.objecter _session_command_op_assign 0 2 2024-09-10T20:34:54.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 20 client.34401.objecter put_session s=0x7f08bc0436c0 osd=0 4 2024-09-10T20:34:54.380 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 10 client.34401.objecter _send_command 2 2024-09-10T20:34:54.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 --> v2:172.21.15.153:6800/4216589932 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f08dc09e690 con 0x7f08bc043d70 2024-09-10T20:34:54.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08d3fff640 1 -- 172.21.15.153:0/2607457677 <== osd.0 v2:172.21.15.153:6800/4216589932 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f08dc09e690 con 0x7f08bc043d70 2024-09-10T20:34:54.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08d3fff640 10 client.34401.objecter ms_dispatch 0x7f08dc000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:34:54.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08d3fff640 10 client.34401.objecter _finish_command 2 = system:0 2024-09-10T20:34:54.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08d3fff640 20 client.34401.objecter put_session s=0x7f08bc0436c0 osd=0 4 2024-09-10T20:34:54.381 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08d3fff640 15 client.34401.objecter _session_command_op_remove 0 2 2024-09-10T20:34:54.381 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:34:54.381 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:34:54.381 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:34:54.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 10 librados: watch_flush enter 2024-09-10T20:34:54.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 10 librados: watch_flush exit 2024-09-10T20:34:54.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 10 client.34401.objecter close_session for osd.0 2024-09-10T20:34:54.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 >> v2:172.21.15.153:6800/4216589932 conn(0x7f08bc043d70 msgr2=0x7f08bc046210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.383 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 --2- 172.21.15.153:0/2607457677 >> v2:172.21.15.153:6800/4216589932 conn(0x7f08bc043d70 0x7f08bc046210 crc :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 20 client.34401.objecter put_session s=0x7f08bc0436c0 osd=0 1 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 20 client.34401.objecter shutdown clearing up homeless session... 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 10 client.34401.objecter successfully canceled tick 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 >> v2:172.21.15.176:6812/3003473398 conn(0x7f08bc03cea0 msgr2=0x7f08bc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 --2- 172.21.15.153:0/2607457677 >> v2:172.21.15.176:6812/3003473398 conn(0x7f08bc03cea0 0x7f08bc03f360 secure :-1 s=READY pgs=3376 cs=0 l=1 rev1=1 crypto rx=0x7f08d40ab000 tx=0x7f08d40a7070 comp rx=0 tx=0).stop 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 10 monclient: shutdown 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09ebd0 msgr2=0x7f08dc06f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 --2- 172.21.15.153:0/2607457677 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f08dc09ebd0 0x7f08dc06f0e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f08c400c7f0 tx=0x7f08c400ccb0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08db7fe640 1 -- 172.21.15.153:0/2607457677 reap_dead start 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 shutdown_connections 2024-09-10T20:34:54.384 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 >> 172.21.15.153:0/2607457677 conn(0x7f08dc0a7900 msgr2=0x7f08dc134b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 shutdown_connections 2024-09-10T20:34:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 -- 172.21.15.153:0/2607457677 wait complete. 2024-09-10T20:34:54.385 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.377+0000 7f08e1a0e640 1 librados: shutdown 2024-09-10T20:34:54.396 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-10T20:34:54.512 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.509+0000 7ff25c056640 10 monclient: get_monmap_and_config 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.509+0000 7ff25c056640 10 monclient: build_initial_monmap 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.509+0000 7ff25c056640 10 monclient: monmap: 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:54.512646+0000 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:54.512646+0000 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:54.513 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:54.514 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.509+0000 7ff25c056640 1 Processor -- start 2024-09-10T20:34:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.509+0000 7ff25c056640 1 -- start start 2024-09-10T20:34:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.509+0000 7ff25c056640 10 monclient: init 2024-09-10T20:34:54.515 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 10 monclient: _add_conns ranks=[2,1,0,3,4] 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2540a37d0 0x7ff2540a3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 10 monclient(hunting): picked mon.noname-b con 0x7ff2540a37d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff25409cfb0 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff25409fa10 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff254136480 con 0x7ff25409fa10 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff25409d430 con 0x7ff25409cfb0 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff25409eb20 con 0x7ff2540a37d0 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25c056640 10 monclient(hunting): authenticate will time out at 1454.652298s 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2540a37d0 0x7ff2540a3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.516 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25a5f3640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7ff25409fa10 0x7ff254135d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60728/0 (socket says 172.21.15.153:60728) 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2540a37d0 0x7ff2540a3bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37428/0 (socket says 172.21.15.153:37428) 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff25a5f3640 1 -- 172.21.15.153:0/3557617240 learned_addr learned my addr 172.21.15.153:0/3557617240 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient(hunting): get_auth_request con 0x7ff2540a37d0 auth_method 0 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient(hunting): handle_auth_done global_id 54143 payload 274 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 1 -- 172.21.15.153:0/3557617240 >> v1:172.21.15.176:6789/0 conn(0x7ff25409cfb0 legacy=0x7ff254135660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 1 -- 172.21.15.153:0/3557617240 >> v1:172.21.15.153:6789/0 conn(0x7ff25409fa10 legacy=0x7ff254135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient: _finish_hunting 0 2024-09-10T20:34:54.517 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 1 monclient: found mon.noname-b 2024-09-10T20:34:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.513+0000 7ff259df2640 1 -- 172.21.15.153:0/3557617240 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2540a5230 con 0x7ff2540a37d0 2024-09-10T20:34:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 1 -- 172.21.15.153:0/3557617240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2540a37d0 msgr2=0x7ff2540a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:54.518 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 1 --2- 172.21.15.153:0/3557617240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2540a37d0 0x7ff2540a3bb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff250009dc0 tx=0x7ff25001c8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4f1a8f8d755be4e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 1 -- 172.21.15.153:0/3557617240 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2500b2070 con 0x7ff2540a37d0 2024-09-10T20:34:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:54.522 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 10 monclient: dump: 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 10 monclient: _finish_auth 0 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 10 monclient: _check_auth_tickets 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:54.523 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 1 -- 172.21.15.153:0/3557617240 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2500a5ac0 con 0x7ff2540a37d0 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 5 monclient: authenticate success, global_id 54143 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 4 set_mon_vals no callback set 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 1 -- 172.21.15.153:0/3557617240 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2500ad440 con 0x7ff2540a37d0 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff258df0640 10 monclient: dump: 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.524 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient: shutdown 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 -- 172.21.15.153:0/3557617240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2540a37d0 msgr2=0x7ff2540a3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 --2- 172.21.15.153:0/3557617240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2540a37d0 0x7ff2540a3bb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff250009dc0 tx=0x7ff25001c8e0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 -- 172.21.15.153:0/3557617240 shutdown_connections 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 --2- 172.21.15.153:0/3557617240 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff2540a37d0 0x7ff2540a3bb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 -- 172.21.15.153:0/3557617240 >> 172.21.15.153:0/3557617240 conn(0x7ff2540a8480 msgr2=0x7ff2540a48c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 -- 172.21.15.153:0/3557617240 shutdown_connections 2024-09-10T20:34:54.525 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 -- 172.21.15.153:0/3557617240 wait complete. 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient: build_initial_monmap 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient: monmap: 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:54.526 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 librados: starting msgr at 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 librados: starting objecter 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 Processor -- start 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 -- start start 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 librados: setting wanted keys 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 librados: calling monclient init 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient: init 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25409cfb0 0x7ff254073840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient(hunting): picked mon.noname-c con 0x7ff25409cfb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff25409fa10 0x7ff25411e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient(hunting): picked mon.noname-b con 0x7ff25409fa10 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:54.527 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2540a37d0 0x7ff25411e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient(hunting): picked mon.noname-a con 0x7ff2540a37d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff254076db0 con 0x7ff2540a37d0 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff254076c30 con 0x7ff25409fa10 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff254076f30 con 0x7ff25409cfb0 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25c056640 10 monclient(hunting): authenticate will time out at 1454.659713s 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2540a37d0 0x7ff25411e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25409cfb0 0x7ff254073840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2540a37d0 0x7ff25411e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56218/0 (socket says 172.21.15.153:56218) 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 1 -- 172.21.15.153:0/2932591242 learned_addr learned my addr 172.21.15.153:0/2932591242 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:54.528 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 10 monclient(hunting): get_auth_request con 0x7ff2540a37d0 auth_method 0 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 10 monclient(hunting): get_auth_request con 0x7ff25409cfb0 auth_method 0 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff25a5f3640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.517+0000 7ff259df2640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.529 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff2595f1640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff25409fa10 0x7ff25411e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 10 monclient(hunting): handle_auth_done global_id 54146 payload 824 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 1 -- 172.21.15.153:0/2932591242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff25409fa10 msgr2=0x7ff25411e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff25409fa10 0x7ff25411e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 1 -- 172.21.15.153:0/2932591242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2540a37d0 msgr2=0x7ff25411e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2540a37d0 0x7ff25411e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 10 monclient: _finish_hunting 0 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 1 monclient: found mon.noname-c 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 1 -- 172.21.15.153:0/2932591242 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff254120ab0 con 0x7ff25409cfb0 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff2595f1640 10 monclient: get_auth_request con 0x7ff25409fa10 auth_method 0 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25a5f3640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2540a37d0 0x7ff25411e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:54.530 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff2595f1640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff25409fa10 0x7ff25411e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:54.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 1 -- 172.21.15.153:0/2932591242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25409cfb0 msgr2=0x7ff254073840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:54.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff259df2640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25409cfb0 0x7ff254073840 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff2500af000 tx=0x7ff250019680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 client.?.objecter ms_handle_connect 0x7ff25409cfb0 2024-09-10T20:34:54.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:54.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 1 -- 172.21.15.153:0/2932591242 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2500bd020 con 0x7ff25409cfb0 2024-09-10T20:34:54.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.533 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 monclient: dump: 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 monclient: _finish_auth 0 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:34:54.534 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 1 -- 172.21.15.153:0/2932591242 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2500b3040 con 0x7ff25409cfb0 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 1 -- 172.21.15.153:0/2932591242 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2500b2070 con 0x7ff25409cfb0 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 10 monclient: dump: 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.535 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 5 monclient: authenticate success, global_id 54146 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 10 monclient: _renew_subs 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24bfff640 4 set_mon_vals no callback set 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff254121920 con 0x7ff25409cfb0 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 10 client.54146.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 10 monclient: _renew_subs 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff254121640 con 0x7ff25409cfb0 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 1 librados: init done 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:54.536 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2540a3bb0 con 0x7ff25409cfb0 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.521+0000 7ff24a7fc640 1 -- 172.21.15.153:0/2932591242 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7ff2500c0020 con 0x7ff25409cfb0 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff24a7fc640 1 --2- 172.21.15.153:0/2932591242 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff23803ced0 0x7ff23803f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff24a7fc640 1 -- 172.21.15.153:0/2932591242 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7ff2500ea8d0 con 0x7ff25409cfb0 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff24a7fc640 10 client.54146.objecter ms_dispatch 0x7ff254000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff24a7fc640 3 client.54146.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff24a7fc640 3 client.54146.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff24a7fc640 20 client.54146.objecter dump_active .. 0 homeless 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff2595f1640 1 --2- 172.21.15.153:0/2932591242 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff23803ced0 0x7ff23803f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff2595f1640 10 monclient: get_auth_request con 0x7ff23803ced0 auth_method 0 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff24a7fc640 10 client.54146.objecter ms_handle_connect 0x7ff23803ced0 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.525+0000 7ff2595f1640 1 --2- 172.21.15.153:0/2932591242 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff23803ced0 0x7ff23803f390 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7ff2541200b0 tx=0x7ff244009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.537 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.529+0000 7ff24a7fc640 1 -- 172.21.15.153:0/2932591242 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7ff2500c7070 con 0x7ff25409cfb0 2024-09-10T20:34:54.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.529+0000 7ff24a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:54.538 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.529+0000 7ff24a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7ff254135420 con 0x7ff23803ced0 2024-09-10T20:34:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff24a7fc640 1 -- 172.21.15.153:0/2932591242 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff254135420 con 0x7ff23803ced0 2024-09-10T20:34:54.688 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 0 to deep-scrub 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 10 librados: watch_flush enter 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 10 librados: watch_flush exit 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 20 client.54146.objecter shutdown clearing up homeless session... 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 10 client.54146.objecter successfully canceled tick 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff23803ced0 msgr2=0x7ff23803f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 --2- 172.21.15.153:0/2932591242 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff23803ced0 0x7ff23803f390 secure :-1 s=READY pgs=3377 cs=0 l=1 rev1=1 crypto rx=0x7ff2541200b0 tx=0x7ff244009040 comp rx=0 tx=0).stop 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 10 monclient: shutdown 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25409cfb0 msgr2=0x7ff254073840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.689 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25409cfb0 0x7ff254073840 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff2500af000 tx=0x7ff250019680 comp rx=0 tx=0).stop 2024-09-10T20:34:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 shutdown_connections 2024-09-10T20:34:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff2540a37d0 0x7ff25411e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff25409fa10 0x7ff25411e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 --2- 172.21.15.153:0/2932591242 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff23803ced0 0x7ff23803f390 unknown :-1 s=CLOSED pgs=3377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 --2- 172.21.15.153:0/2932591242 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff25409cfb0 0x7ff254073840 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 >> 172.21.15.153:0/2932591242 conn(0x7ff2540a8480 msgr2=0x7ff25413d1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 shutdown_connections 2024-09-10T20:34:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 -- 172.21.15.153:0/2932591242 wait complete. 2024-09-10T20:34:54.690 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.685+0000 7ff25c056640 1 librados: shutdown 2024-09-10T20:34:54.704 INFO:tasks.ceph:Scrubbing osd.1 2024-09-10T20:34:54.704 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:34:54.828 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.825+0000 7fbd709ae640 10 monclient: get_monmap_and_config 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.825+0000 7fbd709ae640 10 monclient: build_initial_monmap 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.825+0000 7fbd709ae640 10 monclient: monmap: 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:54.828615+0000 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:54.828615+0000 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:54.829 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:54.830 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:54.830 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.825+0000 7fbd709ae640 1 Processor -- start 2024-09-10T20:34:54.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.825+0000 7fbd709ae640 1 -- start start 2024-09-10T20:34:54.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.825+0000 7fbd709ae640 10 monclient: init 2024-09-10T20:34:54.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:54.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 10 monclient: _add_conns ranks=[3,1,4,2,0] 2024-09-10T20:34:54.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbd6c0a3f50 0x7fbd6c132d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.832 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 10 monclient(hunting): picked mon.noname-a con 0x7fbd6c0a3f50 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd6c074500 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fbd6c074950 0x7fbd6c1367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 10 monclient(hunting): picked mon.noname-c con 0x7fbd6c074950 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd6c074d70 con 0x7fbd6c074500 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fbd6c0a09f0 con 0x7fbd6c0a3f50 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fbd6c0a4330 con 0x7fbd6c074950 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd709ae640 10 monclient(hunting): authenticate will time out at 1454.968256s 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbd6c0a3f50 0x7fbd6c132d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fbd6c0a3f50 0x7fbd6c132d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56222/0 (socket says 172.21.15.153:56222) 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 1 -- 172.21.15.153:0/3947652833 learned_addr learned my addr 172.21.15.153:0/3947652833 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:54.833 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient(hunting): get_auth_request con 0x7fbd6c0a3f50 auth_method 0 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6ad9d640 1 --2- 172.21.15.153:0/3947652833 >> v2:172.21.15.176:3300/0 conn(0x7fbd6c074950 0x7fbd6c1367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6ad9d640 10 monclient(hunting): get_auth_request con 0x7fbd6c074950 auth_method 0 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6ad9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6ad9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6ad9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient(hunting): handle_auth_done global_id 34407 payload 274 2024-09-10T20:34:54.834 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 1 -- 172.21.15.153:0/3947652833 >> v2:172.21.15.176:3300/0 conn(0x7fbd6c074950 msgr2=0x7fbd6c1367d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 1 --2- 172.21.15.153:0/3947652833 >> v2:172.21.15.176:3300/0 conn(0x7fbd6c074950 0x7fbd6c1367d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 1 -- 172.21.15.153:0/3947652833 >> v1:172.21.15.176:6789/0 conn(0x7fbd6c074500 legacy=0x7fbd6c1360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient: _finish_hunting 0 2024-09-10T20:34:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 1 monclient: found mon.noname-a 2024-09-10T20:34:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.835 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.829+0000 7fbd6a59c640 1 -- 172.21.15.153:0/3947652833 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6c0a2dc0 con 0x7fbd6c0a3f50 2024-09-10T20:34:54.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6a59c640 1 -- 172.21.15.153:0/3947652833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c0a3f50 msgr2=0x7fbd6c132d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:54.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6a59c640 1 --2- 172.21.15.153:0/3947652833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c0a3f50 0x7fbd6c132d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0a2a70 tx=0x7fbd6001c260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a49fb7cc513b3b05 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 1 -- 172.21.15.153:0/3947652833 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd600b0070 con 0x7fbd6c0a3f50 2024-09-10T20:34:54.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:54.837 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 10 monclient: dump: 2024-09-10T20:34:54.837 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.838 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.838 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.838 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.838 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.838 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.838 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.838 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.838 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 10 monclient: _finish_auth 0 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 10 monclient: _check_auth_tickets 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 1 -- 172.21.15.153:0/3947652833 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd600ab600 con 0x7fbd6c0a3f50 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 1 -- 172.21.15.153:0/3947652833 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd600ab9d0 con 0x7fbd6c0a3f50 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd6959a640 10 monclient: dump: 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.839 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 5 monclient: authenticate success, global_id 34407 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 4 set_mon_vals no callback set 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 10 monclient: shutdown 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:54.840 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 -- 172.21.15.153:0/3947652833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c0a3f50 msgr2=0x7fbd6c132d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 --2- 172.21.15.153:0/3947652833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c0a3f50 0x7fbd6c132d80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0a2a70 tx=0x7fbd6001c260 comp rx=0 tx=0).stop 2024-09-10T20:34:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 -- 172.21.15.153:0/3947652833 shutdown_connections 2024-09-10T20:34:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 --2- 172.21.15.153:0/3947652833 >> v2:172.21.15.176:3300/0 conn(0x7fbd6c074950 0x7fbd6c1367d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.841 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 --2- 172.21.15.153:0/3947652833 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c0a3f50 0x7fbd6c132d80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 -- 172.21.15.153:0/3947652833 >> 172.21.15.153:0/3947652833 conn(0x7fbd6c0a7900 msgr2=0x7fbd6c0a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 -- 172.21.15.153:0/3947652833 shutdown_connections 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 -- 172.21.15.153:0/3947652833 wait complete. 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 10 monclient: build_initial_monmap 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 10 monclient: monmap: 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:54.842 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:54.843 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 librados: starting msgr at 2024-09-10T20:34:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 librados: starting objecter 2024-09-10T20:34:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 Processor -- start 2024-09-10T20:34:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 -- start start 2024-09-10T20:34:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 librados: setting wanted keys 2024-09-10T20:34:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 1 librados: calling monclient init 2024-09-10T20:34:54.843 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.833+0000 7fbd709ae640 10 monclient: init 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6c074500 0x7fbd6c070950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient(hunting): picked mon.noname-c con 0x7fbd6c074500 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c074950 0x7fbd6c070e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient(hunting): picked mon.noname-a con 0x7fbd6c074950 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd6c0a3f50 0x7fbd6c0713d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient(hunting): picked mon.noname-b con 0x7fbd6c0a3f50 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c136e90 con 0x7fbd6c074950 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.848 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fbd6c136d10 con 0x7fbd6c0a3f50 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fbd6c140c80 con 0x7fbd6c074500 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient(hunting): authenticate will time out at 1454.976363s 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c074950 0x7fbd6c070e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd6a59c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6c074500 0x7fbd6c070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c074950 0x7fbd6c070e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56238/0 (socket says 172.21.15.153:56238) 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 -- 172.21.15.153:0/1130187857 learned_addr learned my addr 172.21.15.153:0/1130187857 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient(hunting): get_auth_request con 0x7fbd6c074950 auth_method 0 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd6a59c640 10 monclient(hunting): get_auth_request con 0x7fbd6c074500 auth_method 0 2024-09-10T20:34:54.849 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd6a59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd6a59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd6a59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd6a59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd6a59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.850 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd6a59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient(hunting): handle_auth_done global_id 34410 payload 824 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd6ad9d640 1 --2- 172.21.15.153:0/1130187857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd6c0a3f50 0x7fbd6c0713d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 -- 172.21.15.153:0/1130187857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6c074500 msgr2=0x7fbd6c070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 --2- 172.21.15.153:0/1130187857 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fbd6c074500 0x7fbd6c070950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 -- 172.21.15.153:0/1130187857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd6c0a3f50 msgr2=0x7fbd6c0713d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 --2- 172.21.15.153:0/1130187857 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fbd6c0a3f50 0x7fbd6c0713d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient: _finish_hunting 0 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 monclient: found mon.noname-a 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 -- 172.21.15.153:0/1130187857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd6c073e00 con 0x7fbd6c074950 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 -- 172.21.15.153:0/1130187857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c074950 msgr2=0x7fbd6c070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd69d9b640 1 --2- 172.21.15.153:0/1130187857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c074950 0x7fbd6c070e90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00c480 tx=0x7fbd5c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.851 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 client.?.objecter ms_handle_connect 0x7fbd6c074950 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 1 -- 172.21.15.153:0/1130187857 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c012a40 con 0x7fbd6c074950 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 monclient: dump: 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.852 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 monclient: _finish_auth 0 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 monclient: _check_auth_tickets 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 5 monclient: authenticate success, global_id 34410 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 1 -- 172.21.15.153:0/1130187857 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd5c017070 con 0x7fbd6c074950 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient: _renew_subs 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 1 -- 172.21.15.153:0/1130187857 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd5c012e40 con 0x7fbd6c074950 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd6c0722c0 con 0x7fbd6c074950 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd68d99640 4 set_mon_vals no callback set 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 client.34410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient: _renew_subs 2024-09-10T20:34:54.853 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd6c0a6900 con 0x7fbd6c074950 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 librados: init done 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd5affd640 10 monclient: dump: 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:54.854 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 client.34410.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 15 client.34410.objecter _session_command_op_assign -1 1 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 20 client.34410.objecter _get_session osd=-1 returning homeless 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 client.34410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.837+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbd6c09e0a0 con 0x7fbd6c074950 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 1 -- 172.21.15.153:0/1130187857 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fbd5c01c460 con 0x7fbd6c074950 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 1 --2- 172.21.15.153:0/1130187857 >> v2:172.21.15.176:6812/3003473398 conn(0x7fbd4c03cea0 0x7fbd4c03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 1 -- 172.21.15.153:0/1130187857 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7fbd5c04f3a0 con 0x7fbd6c074950 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 10 client.34410.objecter ms_dispatch 0x7fbd6c000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 3 client.34410.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 3 client.34410.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 10 client.34410.objecter checking command 1 2024-09-10T20:34:54.855 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 1 --2- 172.21.15.153:0/1130187857 >> v2:172.21.15.153:6804/1715391555 conn(0x7fbd4c043d60 0x7fbd4c046200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 20 client.34410.objecter _get_session s=0x7fbd4c0436b0 osd=1 3 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 20 client.34410.objecter put_session s=0x7fbd4c0436b0 osd=1 3 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 15 client.34410.objecter _session_command_op_remove -1 1 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 20 client.34410.objecter _get_session s=0x7fbd4c0436b0 osd=1 3 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 20 client.34410.objecter get_session s=0x7fbd4c0436b0 osd=1 3 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 15 client.34410.objecter _session_command_op_assign 1 1 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 20 client.34410.objecter put_session s=0x7fbd4c0436b0 osd=1 4 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 10 client.34410.objecter _send_command 1 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 1 -- 172.21.15.153:0/1130187857 --> v2:172.21.15.153:6804/1715391555 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbd4c046890 con 0x7fbd4c043d60 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 20 client.34410.objecter dump_active .. 0 homeless 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 1 -- 172.21.15.153:0/1130187857 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=317) ==== 24+0+0 (secure 0 0 0) 0x7fbd5c056050 con 0x7fbd6c074950 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd5affd640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd6ad9d640 1 --2- 172.21.15.153:0/1130187857 >> v2:172.21.15.153:6804/1715391555 conn(0x7fbd4c043d60 0x7fbd4c046200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd6ad9d640 10 monclient: get_auth_request con 0x7fbd4c043d60 auth_method 0 2024-09-10T20:34:54.856 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd6a59c640 1 --2- 172.21.15.153:0/1130187857 >> v2:172.21.15.176:6812/3003473398 conn(0x7fbd4c03cea0 0x7fbd4c03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.841+0000 7fbd6a59c640 10 monclient: get_auth_request con 0x7fbd4c03cea0 auth_method 0 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.845+0000 7fbd5affd640 10 client.34410.objecter ms_handle_connect 0x7fbd4c03cea0 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.845+0000 7fbd6a59c640 1 --2- 172.21.15.153:0/1130187857 >> v2:172.21.15.176:6812/3003473398 conn(0x7fbd4c03cea0 0x7fbd4c03f360 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0a4bf0 tx=0x7fbd600a6000 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.845+0000 7fbd6ad9d640 1 --2- 172.21.15.153:0/1130187857 >> v2:172.21.15.153:6804/1715391555 conn(0x7fbd4c043d60 0x7fbd4c046200 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.845+0000 7fbd5affd640 10 client.34410.objecter ms_handle_connect 0x7fbd4c043d60 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.845+0000 7fbd5affd640 1 -- 172.21.15.153:0/1130187857 <== osd.1 v2:172.21.15.153:6804/1715391555 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbd4c046890 con 0x7fbd4c043d60 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.845+0000 7fbd5affd640 10 client.34410.objecter ms_dispatch 0x7fbd6c000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.845+0000 7fbd5affd640 10 client.34410.objecter _finish_command 1 = system:0 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.845+0000 7fbd5affd640 20 client.34410.objecter put_session s=0x7fbd4c0436b0 osd=1 4 2024-09-10T20:34:54.857 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.845+0000 7fbd5affd640 15 client.34410.objecter _session_command_op_remove 1 1 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 10 client.34410.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 15 client.34410.objecter _session_command_op_assign -1 2 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 20 client.34410.objecter _get_session s=0x7fbd4c0436b0 osd=1 3 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 20 client.34410.objecter put_session s=0x7fbd4c0436b0 osd=1 3 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 20 client.34410.objecter _get_session s=0x7fbd4c0436b0 osd=1 3 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 15 client.34410.objecter _session_command_op_remove -1 2 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 20 client.34410.objecter get_session s=0x7fbd4c0436b0 osd=1 3 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 15 client.34410.objecter _session_command_op_assign 1 2 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 20 client.34410.objecter put_session s=0x7fbd4c0436b0 osd=1 4 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 10 client.34410.objecter _send_command 2 2024-09-10T20:34:54.861 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 --> v2:172.21.15.153:6804/1715391555 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fbd6c132d80 con 0x7fbd4c043d60 2024-09-10T20:34:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd5affd640 1 -- 172.21.15.153:0/1130187857 <== osd.1 v2:172.21.15.153:6804/1715391555 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fbd6c132d80 con 0x7fbd4c043d60 2024-09-10T20:34:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd5affd640 10 client.34410.objecter ms_dispatch 0x7fbd6c000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:34:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd5affd640 10 client.34410.objecter _finish_command 2 = system:0 2024-09-10T20:34:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd5affd640 20 client.34410.objecter put_session s=0x7fbd4c0436b0 osd=1 4 2024-09-10T20:34:54.862 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd5affd640 15 client.34410.objecter _session_command_op_remove 1 2 2024-09-10T20:34:54.862 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:34:54.862 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:34:54.862 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:34:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 10 librados: watch_flush enter 2024-09-10T20:34:54.863 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 10 librados: watch_flush exit 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 10 client.34410.objecter close_session for osd.1 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 >> v2:172.21.15.153:6804/1715391555 conn(0x7fbd4c043d60 msgr2=0x7fbd4c046200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 --2- 172.21.15.153:0/1130187857 >> v2:172.21.15.153:6804/1715391555 conn(0x7fbd4c043d60 0x7fbd4c046200 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 20 client.34410.objecter put_session s=0x7fbd4c0436b0 osd=1 1 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 20 client.34410.objecter shutdown clearing up homeless session... 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 10 client.34410.objecter successfully canceled tick 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 >> v2:172.21.15.176:6812/3003473398 conn(0x7fbd4c03cea0 msgr2=0x7fbd4c03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 --2- 172.21.15.153:0/1130187857 >> v2:172.21.15.176:6812/3003473398 conn(0x7fbd4c03cea0 0x7fbd4c03f360 secure :-1 s=READY pgs=3378 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c0a4bf0 tx=0x7fbd600a6000 comp rx=0 tx=0).stop 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 10 monclient: shutdown 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c074950 msgr2=0x7fbd6c070e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 --2- 172.21.15.153:0/1130187857 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fbd6c074950 0x7fbd6c070e90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbd5c00c480 tx=0x7fbd5c00c940 comp rx=0 tx=0).stop 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd6ad9d640 1 -- 172.21.15.153:0/1130187857 reap_dead start 2024-09-10T20:34:54.864 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 shutdown_connections 2024-09-10T20:34:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 >> 172.21.15.153:0/1130187857 conn(0x7fbd6c0a7900 msgr2=0x7fbd6c12c580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 shutdown_connections 2024-09-10T20:34:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 -- 172.21.15.153:0/1130187857 wait complete. 2024-09-10T20:34:54.865 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.857+0000 7fbd709ae640 1 librados: shutdown 2024-09-10T20:34:54.877 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-10T20:34:54.990 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient: get_monmap_and_config 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient: build_initial_monmap 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient: monmap: 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:54.990610+0000 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:54.990610+0000 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:54.991 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:54.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 1 Processor -- start 2024-09-10T20:34:54.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 1 -- start start 2024-09-10T20:34:54.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient: init 2024-09-10T20:34:54.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:54.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient: _add_conns ranks=[0,3,4,2,1] 2024-09-10T20:34:54.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5af00a41a0 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:54.997 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f5af00a4580 0x7f5af013d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient(hunting): picked mon.noname-a con 0x7f5af00a4580 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f5af00a49d0 0x7f5af013fd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient(hunting): picked mon.noname-c con 0x7f5af00a49d0 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5af00a6a20 con 0x7f5af00a41a0 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f5af00a2570 con 0x7f5af00a4580 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f5af0070400 con 0x7f5af00a49d0 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5af534c640 10 monclient(hunting): authenticate will time out at 1455.130192s 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aeeffd640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f5af00a41a0 0x7f5af00a3740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60730/0 (socket says 172.21.15.153:60730) 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aeeffd640 1 -- 172.21.15.153:0/3279287664 learned_addr learned my addr 172.21.15.153:0/3279287664 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 1 --2- 172.21.15.153:0/3279287664 >> v2:172.21.15.153:3300/0 conn(0x7f5af00a4580 0x7f5af013d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.998 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient(hunting): get_auth_request con 0x7f5af00a4580 auth_method 0 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aef7fe640 1 --2- 172.21.15.153:0/3279287664 >> v2:172.21.15.176:3300/0 conn(0x7f5af00a49d0 0x7f5af013fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aef7fe640 10 monclient(hunting): get_auth_request con 0x7f5af00a49d0 auth_method 0 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:54.999 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_done global_id 34413 payload 274 2024-09-10T20:34:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 1 -- 172.21.15.153:0/3279287664 >> v2:172.21.15.176:3300/0 conn(0x7f5af00a49d0 msgr2=0x7f5af013fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 1 --2- 172.21.15.153:0/3279287664 >> v2:172.21.15.176:3300/0 conn(0x7f5af00a49d0 0x7f5af013fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 1 -- 172.21.15.153:0/3279287664 >> v1:172.21.15.153:6789/0 conn(0x7f5af00a41a0 legacy=0x7f5af00a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 1 monclient: found mon.noname-a 2024-09-10T20:34:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:55.000 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.989+0000 7f5aee7fc640 1 -- 172.21.15.153:0/3279287664 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af00a85a0 con 0x7f5af00a4580 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 1 -- 172.21.15.153:0/3279287664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a4580 msgr2=0x7f5af013d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 1 --2- 172.21.15.153:0/3279287664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a4580 0x7f5af013d7c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5af0070040 tx=0x7f5ae401c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5dda3427ca0070e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 1 -- 172.21.15.153:0/3279287664 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40b0070 con 0x7f5af00a4580 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 10 monclient: dump: 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.006 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.007 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.007 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.007 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.007 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.007 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.007 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.007 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 5 monclient: authenticate success, global_id 34413 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 1 -- 172.21.15.153:0/3279287664 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae40a9710 con 0x7f5af00a4580 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 1 -- 172.21.15.153:0/3279287664 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae40a9ae0 con 0x7f5af00a4580 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 4 set_mon_vals no callback set 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aedffb640 10 monclient: dump: 2024-09-10T20:34:55.008 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient: shutdown 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 -- 172.21.15.153:0/3279287664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a4580 msgr2=0x7f5af013d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 --2- 172.21.15.153:0/3279287664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a4580 0x7f5af013d7c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5af0070040 tx=0x7f5ae401c340 comp rx=0 tx=0).stop 2024-09-10T20:34:55.009 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 -- 172.21.15.153:0/3279287664 shutdown_connections 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 --2- 172.21.15.153:0/3279287664 >> v2:172.21.15.176:3300/0 conn(0x7f5af00a49d0 0x7f5af013fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 --2- 172.21.15.153:0/3279287664 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a4580 0x7f5af013d7c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 -- 172.21.15.153:0/3279287664 >> 172.21.15.153:0/3279287664 conn(0x7f5af00ab7e0 msgr2=0x7f5af00a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 -- 172.21.15.153:0/3279287664 shutdown_connections 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 -- 172.21.15.153:0/3279287664 wait complete. 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient: build_initial_monmap 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient: monmap: 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.010 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 librados: starting msgr at 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 librados: starting objecter 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 Processor -- start 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 -- start start 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 librados: setting wanted keys 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 librados: calling monclient init 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient: init 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af00a41a0 0x7f5af0128640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.011 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient(hunting): picked mon.noname-b con 0x7f5af00a41a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a49d0 0x7f5af01264e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient(hunting): picked mon.noname-a con 0x7f5af00a49d0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af013f4b0 0x7f5af01d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient(hunting): picked mon.noname-c con 0x7f5af013f4b0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f5af0142e00 con 0x7f5af00a49d0 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f5af0142c80 con 0x7f5af00a41a0 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f5af0142f80 con 0x7f5af013f4b0 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5af534c640 10 monclient(hunting): authenticate will time out at 1455.139061s 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a49d0 0x7f5af01264e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.012 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af013f4b0 0x7f5af01d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a49d0 0x7f5af01264e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56264/0 (socket says 172.21.15.153:56264) 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af013f4b0 0x7f5af01d02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37462/0 (socket says 172.21.15.153:37462) 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 1 -- 172.21.15.153:0/4041725904 learned_addr learned my addr 172.21.15.153:0/4041725904 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 10 monclient(hunting): get_auth_request con 0x7f5af00a49d0 auth_method 0 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient(hunting): get_auth_request con 0x7f5af013f4b0 auth_method 0 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.013 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient(hunting): handle_auth_done global_id 54152 payload 824 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aeeffd640 1 --2- 172.21.15.153:0/4041725904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af00a41a0 0x7f5af0128640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 1 -- 172.21.15.153:0/4041725904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af00a41a0 msgr2=0x7f5af0128640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 1 --2- 172.21.15.153:0/4041725904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af00a41a0 0x7f5af0128640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 1 -- 172.21.15.153:0/4041725904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a49d0 msgr2=0x7f5af01264e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 1 --2- 172.21.15.153:0/4041725904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a49d0 0x7f5af01264e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 1 monclient: found mon.noname-c 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.014 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aef7fe640 1 -- 172.21.15.153:0/4041725904 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5af0127ae0 con 0x7f5af013f4b0 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:54.997+0000 7f5aee7fc640 1 --2- 172.21.15.153:0/4041725904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a49d0 0x7f5af01264e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5aef7fe640 1 -- 172.21.15.153:0/4041725904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af013f4b0 msgr2=0x7f5af01d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5aef7fe640 1 --2- 172.21.15.153:0/4041725904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af013f4b0 0x7f5af01d02c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8007d60 tx=0x7f5ae800e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 client.?.objecter ms_handle_connect 0x7f5af013f4b0 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 1 -- 172.21.15.153:0/4041725904 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae80039a0 con 0x7f5af013f4b0 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 monclient: dump: 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.015 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 5 monclient: authenticate success, global_id 54152 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 10 monclient: _renew_subs 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5af00a21c0 con 0x7f5af013f4b0 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 10 client.54152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 10 monclient: _renew_subs 2024-09-10T20:34:55.016 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5af01295a0 con 0x7f5af013f4b0 2024-09-10T20:34:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 1 librados: init done 2024-09-10T20:34:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 1 -- 172.21.15.153:0/4041725904 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae8003b40 con 0x7f5af013f4b0 2024-09-10T20:34:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 1 -- 172.21.15.153:0/4041725904 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae8004020 con 0x7f5af013f4b0 2024-09-10T20:34:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:55.017 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 monclient: dump: 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.018 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5aecff9640 4 set_mon_vals no callback set 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ad0005090 con 0x7f5af013f4b0 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 1 -- 172.21.15.153:0/4041725904 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f5ae801b020 con 0x7f5af013f4b0 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 1 --2- 172.21.15.153:0/4041725904 >> v2:172.21.15.176:6812/3003473398 conn(0x7f5adc03ced0 0x7f5adc03f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 1 -- 172.21.15.153:0/4041725904 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f5ae804f6d0 con 0x7f5af013f4b0 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 10 client.54152.objecter ms_dispatch 0x7f5af0000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 3 client.54152.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 3 client.54152.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5ad77fe640 20 client.54152.objecter dump_active .. 0 homeless 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5aeeffd640 1 --2- 172.21.15.153:0/4041725904 >> v2:172.21.15.176:6812/3003473398 conn(0x7f5adc03ced0 0x7f5adc03f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.019 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.001+0000 7f5aeeffd640 10 monclient: get_auth_request con 0x7f5adc03ced0 auth_method 0 2024-09-10T20:34:55.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.005+0000 7f5aeeffd640 1 --2- 172.21.15.153:0/4041725904 >> v2:172.21.15.176:6812/3003473398 conn(0x7f5adc03ced0 0x7f5adc03f390 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005060 tx=0x7f5ae0009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.005+0000 7f5ad77fe640 10 client.54152.objecter ms_handle_connect 0x7f5adc03ced0 2024-09-10T20:34:55.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.005+0000 7f5ad77fe640 1 -- 172.21.15.153:0/4041725904 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f5ae8004640 con 0x7f5af013f4b0 2024-09-10T20:34:55.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.005+0000 7f5ad77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.020 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.005+0000 7f5ad77fe640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:55.178 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f5ad0000bd0 con 0x7f5adc03ced0 2024-09-10T20:34:55.179 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5ad77fe640 1 -- 172.21.15.153:0/4041725904 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5ad0000bd0 con 0x7f5adc03ced0 2024-09-10T20:34:55.179 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 1 to deep-scrub 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 10 librados: watch_flush enter 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 10 librados: watch_flush exit 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 20 client.54152.objecter shutdown clearing up homeless session... 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 10 client.54152.objecter successfully canceled tick 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 >> v2:172.21.15.176:6812/3003473398 conn(0x7f5adc03ced0 msgr2=0x7f5adc03f390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 --2- 172.21.15.153:0/4041725904 >> v2:172.21.15.176:6812/3003473398 conn(0x7f5adc03ced0 0x7f5adc03f390 secure :-1 s=READY pgs=3379 cs=0 l=1 rev1=1 crypto rx=0x7f5ae0005060 tx=0x7f5ae0009040 comp rx=0 tx=0).stop 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 10 monclient: shutdown 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af013f4b0 msgr2=0x7f5af01d02c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 --2- 172.21.15.153:0/4041725904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af013f4b0 0x7f5af01d02c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8007d60 tx=0x7f5ae800e4b0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 shutdown_connections 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 --2- 172.21.15.153:0/4041725904 >> v2:172.21.15.176:6812/3003473398 conn(0x7f5adc03ced0 0x7f5adc03f390 unknown :-1 s=CLOSED pgs=3379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.182 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 --2- 172.21.15.153:0/4041725904 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f5af013f4b0 0x7f5af01d02c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 --2- 172.21.15.153:0/4041725904 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f5af00a49d0 0x7f5af01264e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 --2- 172.21.15.153:0/4041725904 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f5af00a41a0 0x7f5af0128640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.177+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 >> 172.21.15.153:0/4041725904 conn(0x7f5af00ab7e0 msgr2=0x7f5af009f540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:55.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.181+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 shutdown_connections 2024-09-10T20:34:55.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.181+0000 7f5af534c640 1 -- 172.21.15.153:0/4041725904 wait complete. 2024-09-10T20:34:55.183 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.181+0000 7f5af534c640 1 librados: shutdown 2024-09-10T20:34:55.198 INFO:tasks.ceph:Scrubbing osd.2 2024-09-10T20:34:55.198 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient: get_monmap_and_config 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient: build_initial_monmap 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient: monmap: 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:55.314840+0000 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:55.314840+0000 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:55.315 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:55.316 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:55.316 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:55.316 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:55.316 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 1 Processor -- start 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 1 -- start start 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient: init 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient: _add_conns ranks=[0,1,4,3,2] 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f06e80a3f50 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f06e8074500 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f06e8074950 0x7f06e813eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient(hunting): picked mon.noname-c con 0x7f06e8074950 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06e8133490 con 0x7f06e80a3f50 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06e8074d70 con 0x7f06e8074500 2024-09-10T20:34:55.318 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f06e80a09f0 con 0x7f06e8074950 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ef2c9640 10 monclient(hunting): authenticate will time out at 1455.454714s 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed065640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f06e80a3f50 0x7f06e8132d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60734/0 (socket says 172.21.15.153:60734) 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed065640 1 -- 172.21.15.153:0/10865875 learned_addr learned my addr 172.21.15.153:0/10865875 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 1 --2- 172.21.15.153:0/10865875 >> v2:172.21.15.176:3300/0 conn(0x7f06e8074950 0x7f06e813eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient(hunting): get_auth_request con 0x7f06e8074950 auth_method 0 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.319 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient(hunting): handle_auth_done global_id 20227 payload 274 2024-09-10T20:34:55.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 1 -- 172.21.15.153:0/10865875 >> v1:172.21.15.176:6789/0 conn(0x7f06e8074500 legacy=0x7f06e813b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 1 -- 172.21.15.153:0/10865875 >> v1:172.21.15.153:6789/0 conn(0x7f06e80a3f50 legacy=0x7f06e8132d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 1 monclient: found mon.noname-c 2024-09-10T20:34:55.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.176:3300/0 2024-09-10T20:34:55.320 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.313+0000 7f06ed866640 1 -- 172.21.15.153:0/10865875 --> v2:172.21.15.176:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e80a2dc0 con 0x7f06e8074950 2024-09-10T20:34:55.321 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 1 -- 172.21.15.153:0/10865875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06e8074950 msgr2=0x7f06e813eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 1 --2- 172.21.15.153:0/10865875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06e8074950 0x7f06e813eb70 secure :-1 s=READY pgs=5773 cs=0 l=1 rev1=1 crypto rx=0x7f06e00097c0 tx=0x7f06e001c210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=221b808feebd3e43 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 1 -- 172.21.15.153:0/10865875 <== mon.1 v2:172.21.15.176:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e00b0070 con 0x7f06e8074950 2024-09-10T20:34:55.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.322 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 10 monclient: dump: 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.323 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 1 -- 172.21.15.153:0/10865875 <== mon.1 v2:172.21.15.176:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06e00ab600 con 0x7f06e8074950 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 1 -- 172.21.15.153:0/10865875 <== mon.1 v2:172.21.15.176:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06e00ab9d0 con 0x7f06e8074950 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06e7fff640 10 monclient: dump: 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.324 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.325 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 5 monclient: authenticate success, global_id 20227 2024-09-10T20:34:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:55.325 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 4 set_mon_vals no callback set 2024-09-10T20:34:55.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient: shutdown 2024-09-10T20:34:55.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:55.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 -- 172.21.15.153:0/10865875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06e8074950 msgr2=0x7f06e813eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 --2- 172.21.15.153:0/10865875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06e8074950 0x7f06e813eb70 secure :-1 s=READY pgs=5773 cs=0 l=1 rev1=1 crypto rx=0x7f06e00097c0 tx=0x7f06e001c210 comp rx=0 tx=0).stop 2024-09-10T20:34:55.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 -- 172.21.15.153:0/10865875 shutdown_connections 2024-09-10T20:34:55.330 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 --2- 172.21.15.153:0/10865875 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06e8074950 0x7f06e813eb70 unknown :-1 s=CLOSED pgs=5773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 -- 172.21.15.153:0/10865875 >> 172.21.15.153:0/10865875 conn(0x7f06e80a7900 msgr2=0x7f06e80a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 -- 172.21.15.153:0/10865875 shutdown_connections 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 -- 172.21.15.153:0/10865875 wait complete. 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient: build_initial_monmap 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient: monmap: 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:55.331 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 librados: starting msgr at 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 librados: starting objecter 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 Processor -- start 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 -- start start 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 librados: setting wanted keys 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 librados: calling monclient init 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient: init 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8074500 0x7f06e813dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient(hunting): picked mon.noname-a con 0x7f06e8074500 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06e8074950 0x7f06e809e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.332 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient(hunting): picked mon.noname-b con 0x7f06e8074950 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a3f50 0x7f06e809e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient(hunting): picked mon.noname-c con 0x7f06e80a3f50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f06e8141cc0 con 0x7f06e8074500 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f06e8141b40 con 0x7f06e8074950 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f06e8141e40 con 0x7f06e80a3f50 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ef2c9640 10 monclient(hunting): authenticate will time out at 1455.463451s 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8074500 0x7f06e813dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8074500 0x7f06e813dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56276/0 (socket says 172.21.15.153:56276) 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 1 -- 172.21.15.153:0/862926464 learned_addr learned my addr 172.21.15.153:0/862926464 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:55.333 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient(hunting): get_auth_request con 0x7f06e8074500 auth_method 0 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 1 --2- 172.21.15.153:0/862926464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a3f50 0x7f06e809e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 10 monclient(hunting): get_auth_request con 0x7f06e80a3f50 auth_method 0 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed866640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.334 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient(hunting): handle_auth_done global_id 34419 payload 824 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 1 -- 172.21.15.153:0/862926464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a3f50 msgr2=0x7f06e809e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ec864640 1 --2- 172.21.15.153:0/862926464 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06e8074950 0x7f06e809e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 1 --2- 172.21.15.153:0/862926464 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f06e80a3f50 0x7f06e809e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 1 -- 172.21.15.153:0/862926464 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06e8074950 msgr2=0x7f06e809e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 1 --2- 172.21.15.153:0/862926464 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f06e8074950 0x7f06e809e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 1 monclient: found mon.noname-a 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.321+0000 7f06ed065640 1 -- 172.21.15.153:0/862926464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06e81306d0 con 0x7f06e8074500 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ed065640 1 -- 172.21.15.153:0/862926464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8074500 msgr2=0x7f06e813dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ed065640 1 --2- 172.21.15.153:0/862926464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8074500 0x7f06e813dbe0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f06d800cc40 tx=0x7f06d800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 client.?.objecter ms_handle_connect 0x7f06e8074500 2024-09-10T20:34:55.335 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 1 -- 172.21.15.153:0/862926464 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8012400 con 0x7f06e8074500 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: dump: 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.336 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 5 monclient: authenticate success, global_id 34419 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 1 -- 172.21.15.153:0/862926464 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d80125a0 con 0x7f06e8074500 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 10 monclient: _renew_subs 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06e8074290 con 0x7f06e8074500 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.337 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 1 -- 172.21.15.153:0/862926464 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8012990 con 0x7f06e8074500 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: dump: 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.338 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e6ffd640 4 set_mon_vals no callback set 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 10 client.34419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 10 monclient: _renew_subs 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06e813bfb0 con 0x7f06e8074500 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 1 librados: init done 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 10 client.34419.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 15 client.34419.objecter _session_command_op_assign -1 1 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 20 client.34419.objecter _get_session osd=-1 returning homeless 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 10 client.34419.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f06e8132d80 con 0x7f06e8074500 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 1 -- 172.21.15.153:0/862926464 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f06d8012bb0 con 0x7f06e8074500 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 1 --2- 172.21.15.153:0/862926464 >> v2:172.21.15.176:6812/3003473398 conn(0x7f06d403cea0 0x7f06d403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.339 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 1 -- 172.21.15.153:0/862926464 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f06d804fd30 con 0x7f06e8074500 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 client.34419.objecter ms_dispatch 0x7f06e8000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 3 client.34419.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 3 client.34419.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 client.34419.objecter checking command 1 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 1 --2- 172.21.15.153:0/862926464 >> v2:172.21.15.153:6808/1626005750 conn(0x7f06d4043d70 0x7f06d4046210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 20 client.34419.objecter _get_session s=0x7f06d40436c0 osd=2 3 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 20 client.34419.objecter put_session s=0x7f06d40436c0 osd=2 3 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 15 client.34419.objecter _session_command_op_remove -1 1 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 20 client.34419.objecter _get_session s=0x7f06d40436c0 osd=2 3 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 20 client.34419.objecter get_session s=0x7f06d40436c0 osd=2 3 2024-09-10T20:34:55.340 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 15 client.34419.objecter _session_command_op_assign 2 1 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 20 client.34419.objecter put_session s=0x7f06d40436c0 osd=2 4 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 client.34419.objecter _send_command 1 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 1 -- 172.21.15.153:0/862926464 --> v2:172.21.15.153:6808/1626005750 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06d8014e50 con 0x7f06d4043d70 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 20 client.34419.objecter dump_active .. 0 homeless 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 1 -- 172.21.15.153:0/862926464 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_get_version_reply(handle=1 version=317) ==== 24+0+0 (secure 0 0 0) 0x7f06d8052380 con 0x7f06e8074500 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06e5ffb640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ed866640 1 --2- 172.21.15.153:0/862926464 >> v2:172.21.15.153:6808/1626005750 conn(0x7f06d4043d70 0x7f06d4046210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ed866640 10 monclient: get_auth_request con 0x7f06d4043d70 auth_method 0 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.325+0000 7f06ec864640 1 --2- 172.21.15.153:0/862926464 >> v2:172.21.15.176:6812/3003473398 conn(0x7f06d403cea0 0x7f06d403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.341 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.329+0000 7f06ec864640 10 monclient: get_auth_request con 0x7f06d403cea0 auth_method 0 2024-09-10T20:34:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.329+0000 7f06ec864640 1 --2- 172.21.15.153:0/862926464 >> v2:172.21.15.176:6812/3003473398 conn(0x7f06d403cea0 0x7f06d403f360 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7f06dc007fd0 tx=0x7f06dc009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.329+0000 7f06e5ffb640 10 client.34419.objecter ms_handle_connect 0x7f06d403cea0 2024-09-10T20:34:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.333+0000 7f06e5ffb640 10 client.34419.objecter ms_handle_connect 0x7f06d4043d70 2024-09-10T20:34:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.333+0000 7f06ed866640 1 --2- 172.21.15.153:0/862926464 >> v2:172.21.15.153:6808/1626005750 conn(0x7f06d4043d70 0x7f06d4046210 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.333+0000 7f06e5ffb640 1 -- 172.21.15.153:0/862926464 <== osd.2 v2:172.21.15.153:6808/1626005750 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06d8014e50 con 0x7f06d4043d70 2024-09-10T20:34:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.333+0000 7f06e5ffb640 10 client.34419.objecter ms_dispatch 0x7f06e8000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:34:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.333+0000 7f06e5ffb640 10 client.34419.objecter _finish_command 1 = system:0 2024-09-10T20:34:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.333+0000 7f06e5ffb640 20 client.34419.objecter put_session s=0x7f06d40436c0 osd=2 4 2024-09-10T20:34:55.342 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.333+0000 7f06e5ffb640 15 client.34419.objecter _session_command_op_remove 2 1 2024-09-10T20:34:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 10 client.34419.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-10T20:34:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 15 client.34419.objecter _session_command_op_assign -1 2 2024-09-10T20:34:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 20 client.34419.objecter _get_session s=0x7f06d40436c0 osd=2 3 2024-09-10T20:34:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 20 client.34419.objecter put_session s=0x7f06d40436c0 osd=2 3 2024-09-10T20:34:55.347 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 20 client.34419.objecter _get_session s=0x7f06d40436c0 osd=2 3 2024-09-10T20:34:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 15 client.34419.objecter _session_command_op_remove -1 2 2024-09-10T20:34:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 20 client.34419.objecter get_session s=0x7f06d40436c0 osd=2 3 2024-09-10T20:34:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 15 client.34419.objecter _session_command_op_assign 2 2 2024-09-10T20:34:55.348 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 20 client.34419.objecter put_session s=0x7f06d40436c0 osd=2 4 2024-09-10T20:34:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 10 client.34419.objecter _send_command 2 2024-09-10T20:34:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 --> v2:172.21.15.153:6808/1626005750 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f06e80a0530 con 0x7f06d4043d70 2024-09-10T20:34:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06e5ffb640 1 -- 172.21.15.153:0/862926464 <== osd.2 v2:172.21.15.153:6808/1626005750 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f06e80a0530 con 0x7f06d4043d70 2024-09-10T20:34:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06e5ffb640 10 client.34419.objecter ms_dispatch 0x7f06e8000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:34:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06e5ffb640 10 client.34419.objecter _finish_command 2 = system:0 2024-09-10T20:34:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06e5ffb640 20 client.34419.objecter put_session s=0x7f06d40436c0 osd=2 4 2024-09-10T20:34:55.349 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06e5ffb640 15 client.34419.objecter _session_command_op_remove 2 2 2024-09-10T20:34:55.350 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:34:55.350 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:34:55.351 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 10 librados: watch_flush enter 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 10 librados: watch_flush exit 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 10 client.34419.objecter close_session for osd.2 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 >> v2:172.21.15.153:6808/1626005750 conn(0x7f06d4043d70 msgr2=0x7f06d4046210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 --2- 172.21.15.153:0/862926464 >> v2:172.21.15.153:6808/1626005750 conn(0x7f06d4043d70 0x7f06d4046210 crc :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 20 client.34419.objecter put_session s=0x7f06d40436c0 osd=2 1 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 20 client.34419.objecter shutdown clearing up homeless session... 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 10 client.34419.objecter successfully canceled tick 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 >> v2:172.21.15.176:6812/3003473398 conn(0x7f06d403cea0 msgr2=0x7f06d403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.352 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 --2- 172.21.15.153:0/862926464 >> v2:172.21.15.176:6812/3003473398 conn(0x7f06d403cea0 0x7f06d403f360 secure :-1 s=READY pgs=3380 cs=0 l=1 rev1=1 crypto rx=0x7f06dc007fd0 tx=0x7f06dc009040 comp rx=0 tx=0).stop 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 10 monclient: shutdown 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8074500 msgr2=0x7f06e813dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 --2- 172.21.15.153:0/862926464 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f06e8074500 0x7f06e813dbe0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f06d800cc40 tx=0x7f06d800a430 comp rx=0 tx=0).stop 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ed866640 1 -- 172.21.15.153:0/862926464 reap_dead start 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 shutdown_connections 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 >> 172.21.15.153:0/862926464 conn(0x7f06e80a7900 msgr2=0x7f06e81361b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 shutdown_connections 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 -- 172.21.15.153:0/862926464 wait complete. 2024-09-10T20:34:55.353 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.345+0000 7f06ef2c9640 1 librados: shutdown 2024-09-10T20:34:55.362 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-10T20:34:55.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient: get_monmap_and_config 2024-09-10T20:34:55.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient: build_initial_monmap 2024-09-10T20:34:55.467 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient: monmap: 2024-09-10T20:34:55.467 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:55.467 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.467 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:55.466485+0000 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:55.466485+0000 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 1 Processor -- start 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 1 -- start start 2024-09-10T20:34:55.468 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient: init 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient: _add_conns ranks=[0,1,2,4,3] 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f65ac09ce10 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f65ac0a0270 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09d900 0x7f65ac13df60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient(hunting): picked mon.noname-b con 0x7f65ac09d900 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65ac0a3040 con 0x7f65ac09ce10 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65ac09ea30 con 0x7f65ac0a0270 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f65ac09f5b0 con 0x7f65ac09d900 2024-09-10T20:34:55.470 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b33f2640 10 monclient(hunting): authenticate will time out at 1455.606086s 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b118e640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7f65ac09ce10 0x7f65ac09d1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60736/0 (socket says 172.21.15.153:60736) 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b118e640 1 -- 172.21.15.153:0/4099379814 learned_addr learned my addr 172.21.15.153:0/4099379814 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 1 --2- 172.21.15.153:0/4099379814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09d900 0x7f65ac13df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient(hunting): get_auth_request con 0x7f65ac09d900 auth_method 0 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient(hunting): handle_auth_done global_id 54158 payload 274 2024-09-10T20:34:55.471 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 1 -- 172.21.15.153:0/4099379814 >> v1:172.21.15.176:6789/0 conn(0x7f65ac0a0270 legacy=0x7f65ac13a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 1 -- 172.21.15.153:0/4099379814 >> v1:172.21.15.153:6789/0 conn(0x7f65ac09ce10 legacy=0x7f65ac09d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 1 monclient: found mon.noname-b 2024-09-10T20:34:55.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.472 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.465+0000 7f65b198f640 1 -- 172.21.15.153:0/4099379814 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ac09f890 con 0x7f65ac09d900 2024-09-10T20:34:55.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b198f640 1 -- 172.21.15.153:0/4099379814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09d900 msgr2=0x7f65ac13df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b198f640 1 --2- 172.21.15.153:0/4099379814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09d900 0x7f65ac13df60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f65a40097c0 tx=0x7f65a401c380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1117ab220da56dd server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 1 -- 172.21.15.153:0/4099379814 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a40b0070 con 0x7f65ac09d900 2024-09-10T20:34:55.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:55.476 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 10 monclient: dump: 2024-09-10T20:34:55.476 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 1 -- 172.21.15.153:0/4099379814 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65a40ab600 con 0x7f65ac09d900 2024-09-10T20:34:55.477 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 5 monclient: authenticate success, global_id 54158 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 1 -- 172.21.15.153:0/4099379814 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65a40ab9d0 con 0x7f65ac09d900 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 4 set_mon_vals no callback set 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65a3fff640 10 monclient: dump: 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.478 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 10 monclient: shutdown 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 1 -- 172.21.15.153:0/4099379814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09d900 msgr2=0x7f65ac13df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 1 --2- 172.21.15.153:0/4099379814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09d900 0x7f65ac13df60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f65a40097c0 tx=0x7f65a401c380 comp rx=0 tx=0).stop 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 1 -- 172.21.15.153:0/4099379814 shutdown_connections 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 1 --2- 172.21.15.153:0/4099379814 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09d900 0x7f65ac13df60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 1 -- 172.21.15.153:0/4099379814 >> 172.21.15.153:0/4099379814 conn(0x7f65ac0a71b0 msgr2=0x7f65ac0a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 1 -- 172.21.15.153:0/4099379814 shutdown_connections 2024-09-10T20:34:55.479 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.469+0000 7f65b33f2640 1 -- 172.21.15.153:0/4099379814 wait complete. 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient: build_initial_monmap 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient: monmap: 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 librados: starting msgr at 2024-09-10T20:34:55.480 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 librados: starting objecter 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 Processor -- start 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 -- start start 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 librados: setting wanted keys 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 librados: calling monclient init 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient: init 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09ce10 0x7f65ac134be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient(hunting): picked mon.noname-c con 0x7f65ac09ce10 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ac09d900 0x7f65ac135120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient(hunting): picked mon.noname-a con 0x7f65ac09d900 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65ac11fd90 0x7f65ac11d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient(hunting): picked mon.noname-b con 0x7f65ac11fd90 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:55.481 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac141050 con 0x7f65ac09d900 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f65ac140ed0 con 0x7f65ac11fd90 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f65ac1411d0 con 0x7f65ac09ce10 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient(hunting): authenticate will time out at 1455.612931s 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ac09d900 0x7f65ac135120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09ce10 0x7f65ac134be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ac09d900 0x7f65ac135120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56290/0 (socket says 172.21.15.153:56290) 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09ce10 0x7f65ac134be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37482/0 (socket says 172.21.15.153:37482) 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 -- 172.21.15.153:0/1912268524 learned_addr learned my addr 172.21.15.153:0/1912268524 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:55.482 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient(hunting): get_auth_request con 0x7f65ac09d900 auth_method 0 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 10 monclient(hunting): get_auth_request con 0x7f65ac09ce10 auth_method 0 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.483 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient(hunting): handle_auth_done global_id 34422 payload 824 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 -- 172.21.15.153:0/1912268524 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09ce10 msgr2=0x7f65ac134be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b198f640 1 --2- 172.21.15.153:0/1912268524 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65ac11fd90 0x7f65ac11d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 --2- 172.21.15.153:0/1912268524 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09ce10 0x7f65ac134be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 -- 172.21.15.153:0/1912268524 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65ac11fd90 msgr2=0x7f65ac11d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 --2- 172.21.15.153:0/1912268524 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65ac11fd90 0x7f65ac11d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 monclient: found mon.noname-a 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:55.484 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 -- 172.21.15.153:0/1912268524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65ac11f230 con 0x7f65ac09d900 2024-09-10T20:34:55.485 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b118e640 1 --2- 172.21.15.153:0/1912268524 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09ce10 0x7f65ac134be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 -- 172.21.15.153:0/1912268524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ac09d900 msgr2=0x7f65ac135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b098d640 1 --2- 172.21.15.153:0/1912268524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ac09d900 0x7f65ac135120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f659c00c410 tx=0x7f659c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 client.?.objecter ms_handle_connect 0x7f65ac09d900 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 1 -- 172.21.15.153:0/1912268524 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c0119a0 con 0x7f65ac09d900 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 monclient: dump: 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.487 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 1 -- 172.21.15.153:0/1912268524 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f659c017070 con 0x7f65ac09d900 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 5 monclient: authenticate success, global_id 34422 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient: _renew_subs 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:55.488 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65ac0a5cd0 con 0x7f65ac09d900 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 1 -- 172.21.15.153:0/1912268524 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659c011d80 con 0x7f65ac09d900 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a1ffb640 10 monclient: dump: 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.489 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65a37fe640 4 set_mon_vals no callback set 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 client.34422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient: _renew_subs 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65ac121590 con 0x7f65ac09d900 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.473+0000 7f65b33f2640 1 librados: init done 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65b33f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65b33f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65b33f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65ac09e0a0 con 0x7f65ac09d900 2024-09-10T20:34:55.490 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65a1ffb640 1 -- 172.21.15.153:0/1912268524 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f659c01c460 con 0x7f65ac09d900 2024-09-10T20:34:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65a1ffb640 1 --2- 172.21.15.153:0/1912268524 >> v2:172.21.15.176:6812/3003473398 conn(0x7f659003cea0 0x7f659003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65a1ffb640 1 -- 172.21.15.153:0/1912268524 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f659c04f250 con 0x7f65ac09d900 2024-09-10T20:34:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65a1ffb640 10 client.34422.objecter ms_dispatch 0x7f65ac000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:55.491 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65a1ffb640 3 client.34422.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65a1ffb640 3 client.34422.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65a1ffb640 20 client.34422.objecter dump_active .. 0 homeless 2024-09-10T20:34:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65b118e640 1 --2- 172.21.15.153:0/1912268524 >> v2:172.21.15.176:6812/3003473398 conn(0x7f659003cea0 0x7f659003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.477+0000 7f65b118e640 10 monclient: get_auth_request con 0x7f659003cea0 auth_method 0 2024-09-10T20:34:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.481+0000 7f65b118e640 1 --2- 172.21.15.153:0/1912268524 >> v2:172.21.15.176:6812/3003473398 conn(0x7f659003cea0 0x7f659003f360 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7f65ac1359f0 tx=0x7f65a8009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.481+0000 7f65a1ffb640 10 client.34422.objecter ms_handle_connect 0x7f659003cea0 2024-09-10T20:34:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.481+0000 7f65a1ffb640 1 -- 172.21.15.153:0/1912268524 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f659c021020 con 0x7f65ac09d900 2024-09-10T20:34:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.481+0000 7f65a1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.492 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.481+0000 7f65a1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:55.639 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.637+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f65ac09d1f0 con 0x7f659003cea0 2024-09-10T20:34:55.640 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.637+0000 7f65a1ffb640 1 -- 172.21.15.153:0/1912268524 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f65ac09d1f0 con 0x7f659003cea0 2024-09-10T20:34:55.640 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 2 to deep-scrub 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 10 librados: watch_flush enter 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 10 librados: watch_flush exit 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 20 client.34422.objecter shutdown clearing up homeless session... 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 10 client.34422.objecter successfully canceled tick 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 >> v2:172.21.15.176:6812/3003473398 conn(0x7f659003cea0 msgr2=0x7f659003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 --2- 172.21.15.153:0/1912268524 >> v2:172.21.15.176:6812/3003473398 conn(0x7f659003cea0 0x7f659003f360 secure :-1 s=READY pgs=3381 cs=0 l=1 rev1=1 crypto rx=0x7f65ac1359f0 tx=0x7f65a8009040 comp rx=0 tx=0).stop 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 10 monclient: shutdown 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ac09d900 msgr2=0x7f65ac135120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 --2- 172.21.15.153:0/1912268524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ac09d900 0x7f65ac135120 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f659c00c410 tx=0x7f659c00c8d0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 shutdown_connections 2024-09-10T20:34:55.643 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 --2- 172.21.15.153:0/1912268524 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f65ac11fd90 0x7f65ac11d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 --2- 172.21.15.153:0/1912268524 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f65ac09d900 0x7f65ac135120 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 --2- 172.21.15.153:0/1912268524 >> v2:172.21.15.176:6812/3003473398 conn(0x7f659003cea0 0x7f659003f360 unknown :-1 s=CLOSED pgs=3381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 --2- 172.21.15.153:0/1912268524 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f65ac09ce10 0x7f65ac134be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 >> 172.21.15.153:0/1912268524 conn(0x7f65ac0a71b0 msgr2=0x7f65ac13d430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:55.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 shutdown_connections 2024-09-10T20:34:55.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 -- 172.21.15.153:0/1912268524 wait complete. 2024-09-10T20:34:55.644 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.641+0000 7f65b33f2640 1 librados: shutdown 2024-09-10T20:34:55.658 INFO:tasks.ceph:Scrubbing osd.3 2024-09-10T20:34:55.658 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:34:55.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient: get_monmap_and_config 2024-09-10T20:34:55.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient: build_initial_monmap 2024-09-10T20:34:55.787 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient: monmap: 2024-09-10T20:34:55.787 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:55.787 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.788 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:55.786020+0000 2024-09-10T20:34:55.788 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:55.786020+0000 2024-09-10T20:34:55.788 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:55.788 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.788 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:55.789 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:55.789 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:55.789 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:55.789 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:55.789 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 1 Processor -- start 2024-09-10T20:34:55.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 1 -- start start 2024-09-10T20:34:55.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient: init 2024-09-10T20:34:55.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:55.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient: _add_conns ranks=[1,2,3,0,4] 2024-09-10T20:34:55.803 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ffa280a60e0 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a4ff0 0x7ffa2813b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient(hunting): picked mon.noname-b con 0x7ffa280a4ff0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffa280a53d0 0x7ffa2813de20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient(hunting): picked mon.noname-a con 0x7ffa280a53d0 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa280a0d60 con 0x7ffa280a60e0 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7ffa280a0710 con 0x7ffa280a53d0 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffa280a64c0 con 0x7ffa280a4ff0 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2fd84640 10 monclient(hunting): authenticate will time out at 1455.925681s 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a4ff0 0x7ffa2813b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.804 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffa280a53d0 0x7ffa2813de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ffa280a53d0 0x7ffa2813de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56296/0 (socket says 172.21.15.153:56296) 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a4ff0 0x7ffa2813b8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37486/0 (socket says 172.21.15.153:37486) 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 1 -- 172.21.15.153:0/2408169494 learned_addr learned my addr 172.21.15.153:0/2408169494 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 10 monclient(hunting): get_auth_request con 0x7ffa280a53d0 auth_method 0 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient(hunting): get_auth_request con 0x7ffa280a4ff0 auth_method 0 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.805 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient(hunting): handle_auth_done global_id 54164 payload 274 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 1 -- 172.21.15.153:0/2408169494 >> v2:172.21.15.153:3300/0 conn(0x7ffa280a53d0 msgr2=0x7ffa2813de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 1 --2- 172.21.15.153:0/2408169494 >> v2:172.21.15.153:3300/0 conn(0x7ffa280a53d0 0x7ffa2813de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 1 -- 172.21.15.153:0/2408169494 >> v1:172.21.15.176:6789/0 conn(0x7ffa280a60e0 legacy=0x7ffa280a48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 1 monclient: found mon.noname-b 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2d31f640 1 -- 172.21.15.153:0/2408169494 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa280a26a0 con 0x7ffa280a4ff0 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.785+0000 7ffa2e321640 1 --2- 172.21.15.153:0/2408169494 >> v2:172.21.15.153:3300/0 conn(0x7ffa280a53d0 0x7ffa2813de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:55.806 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2d31f640 1 -- 172.21.15.153:0/2408169494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a4ff0 msgr2=0x7ffa2813b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2d31f640 1 --2- 172.21.15.153:0/2408169494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a4ff0 0x7ffa2813b8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffa180097c0 tx=0x7ffa1801c210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29576c4e2fb1e640 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 1 -- 172.21.15.153:0/2408169494 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa180b0070 con 0x7ffa280a4ff0 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 10 monclient: dump: 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.807 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 1 -- 172.21.15.153:0/2408169494 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa180ab600 con 0x7ffa280a4ff0 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 1 -- 172.21.15.153:0/2408169494 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa180ab9d0 con 0x7ffa280a4ff0 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 5 monclient: authenticate success, global_id 54164 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 4 set_mon_vals no callback set 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2cb1e640 10 monclient: dump: 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.808 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 10 monclient: shutdown 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2408169494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a4ff0 msgr2=0x7ffa2813b8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 1 --2- 172.21.15.153:0/2408169494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a4ff0 0x7ffa2813b8d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7ffa180097c0 tx=0x7ffa1801c210 comp rx=0 tx=0).stop 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2408169494 shutdown_connections 2024-09-10T20:34:55.809 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 1 --2- 172.21.15.153:0/2408169494 >> v2:172.21.15.153:3300/0 conn(0x7ffa280a53d0 0x7ffa2813de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 1 --2- 172.21.15.153:0/2408169494 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a4ff0 0x7ffa2813b8d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2408169494 >> 172.21.15.153:0/2408169494 conn(0x7ffa280a9620 msgr2=0x7ffa280a59d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2408169494 shutdown_connections 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.789+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2408169494 wait complete. 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient: build_initial_monmap 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient: monmap: 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:55.810 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 librados: starting msgr at 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 librados: starting objecter 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 Processor -- start 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 -- start start 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 librados: setting wanted keys 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 librados: calling monclient init 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient: init 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa280a4ff0 0x7ffa281d0260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient(hunting): picked mon.noname-a con 0x7ffa280a4ff0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa280a53d0 0x7ffa281d0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.811 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient(hunting): picked mon.noname-b con 0x7ffa280a53d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a60e0 0x7ffa2806ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient(hunting): picked mon.noname-c con 0x7ffa280a60e0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28140f10 con 0x7ffa280a4ff0 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ffa28140d90 con 0x7ffa280a53d0 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ffa28141090 con 0x7ffa280a60e0 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2fd84640 10 monclient(hunting): authenticate will time out at 1455.933021s 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa280a4ff0 0x7ffa281d0260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a60e0 0x7ffa2806ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa280a4ff0 0x7ffa281d0260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56304/0 (socket says 172.21.15.153:56304) 2024-09-10T20:34:55.812 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 1 -- 172.21.15.153:0/2463816948 learned_addr learned my addr 172.21.15.153:0/2463816948 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 10 monclient(hunting): get_auth_request con 0x7ffa280a4ff0 auth_method 0 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient(hunting): get_auth_request con 0x7ffa280a60e0 auth_method 0 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.813 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2d31f640 1 --2- 172.21.15.153:0/2463816948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa280a53d0 0x7ffa281d0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2d31f640 10 monclient(hunting): get_auth_request con 0x7ffa280a53d0 auth_method 0 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2d31f640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2d31f640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2d31f640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient(hunting): handle_auth_done global_id 54167 payload 824 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 1 -- 172.21.15.153:0/2463816948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa280a53d0 msgr2=0x7ffa281d0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 1 --2- 172.21.15.153:0/2463816948 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ffa280a53d0 0x7ffa281d0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 1 -- 172.21.15.153:0/2463816948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa280a4ff0 msgr2=0x7ffa281d0260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 1 --2- 172.21.15.153:0/2463816948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa280a4ff0 0x7ffa281d0260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 1 monclient: found mon.noname-c 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.814 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 1 -- 172.21.15.153:0/2463816948 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2813d280 con 0x7ffa280a60e0 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2db20640 1 --2- 172.21.15.153:0/2463816948 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ffa280a4ff0 0x7ffa281d0260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 1 -- 172.21.15.153:0/2463816948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a60e0 msgr2=0x7ffa2806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa2e321640 1 --2- 172.21.15.153:0/2463816948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a60e0 0x7ffa2806ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400e880 tx=0x7ffa2400ed40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa1e7fc640 10 client.?.objecter ms_handle_connect 0x7ffa280a60e0 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa1e7fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2463816948 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24017070 con 0x7ffa280a60e0 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa1e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa1e7fc640 10 monclient: dump: 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.815 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa1e7fc640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa1e7fc640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.793+0000 7ffa1e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2463816948 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa2400cb50 con 0x7ffa280a60e0 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 5 monclient: authenticate success, global_id 54167 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 10 monclient: _renew_subs 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa280706c0 con 0x7ffa280a60e0 2024-09-10T20:34:55.816 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2463816948 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24004020 con 0x7ffa280a60e0 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 10 monclient: dump: 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.817 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1ffff640 4 set_mon_vals no callback set 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 10 client.54167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 10 monclient: _renew_subs 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2813b740 con 0x7ffa280a60e0 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 1 librados: init done 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2463816948 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7ffa240041c0 con 0x7ffa280a60e0 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 10 client.54167.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 15 client.54167.objecter _session_command_op_assign -1 1 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 20 client.54167.objecter _get_session osd=-1 returning homeless 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 10 client.54167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffa280a48e0 con 0x7ffa280a60e0 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 1 --2- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6812/3003473398 conn(0x7ffa0c03cef0 0x7ffa0c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.818 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2463816948 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7ffa2404f740 con 0x7ffa280a60e0 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 10 client.54167.objecter ms_dispatch 0x7ffa28000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 3 client.54167.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 3 client.54167.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 10 client.54167.objecter checking command 1 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 1 --2- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6808/2953083416 conn(0x7ffa0c043e40 0x7ffa0c0462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 20 client.54167.objecter _get_session s=0x7ffa0c043790 osd=3 3 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 20 client.54167.objecter put_session s=0x7ffa0c043790 osd=3 3 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 15 client.54167.objecter _session_command_op_remove -1 1 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 20 client.54167.objecter _get_session s=0x7ffa0c043790 osd=3 3 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 20 client.54167.objecter get_session s=0x7ffa0c043790 osd=3 3 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 15 client.54167.objecter _session_command_op_assign 3 1 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 20 client.54167.objecter put_session s=0x7ffa0c043790 osd=3 4 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 10 client.54167.objecter _send_command 1 2024-09-10T20:34:55.819 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2463816948 --> v2:172.21.15.176:6808/2953083416 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa2401ce50 con 0x7ffa0c043e40 2024-09-10T20:34:55.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 20 client.54167.objecter dump_active .. 0 homeless 2024-09-10T20:34:55.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2463816948 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=317) ==== 24+0+0 (secure 0 0 0) 0x7ffa24056050 con 0x7ffa280a60e0 2024-09-10T20:34:55.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa1e7fc640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-09-10T20:34:55.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2db20640 1 --2- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6812/3003473398 conn(0x7ffa0c03cef0 0x7ffa0c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2db20640 10 monclient: get_auth_request con 0x7ffa0c03cef0 auth_method 0 2024-09-10T20:34:55.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2d31f640 1 --2- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6808/2953083416 conn(0x7ffa0c043e40 0x7ffa0c0462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.797+0000 7ffa2d31f640 10 monclient: get_auth_request con 0x7ffa0c043e40 auth_method 0 2024-09-10T20:34:55.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.801+0000 7ffa2db20640 1 --2- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6812/3003473398 conn(0x7ffa0c03cef0 0x7ffa0c03f3b0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000f3c0 tx=0x7ffa20003e70 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.820 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.801+0000 7ffa1e7fc640 10 client.54167.objecter ms_handle_connect 0x7ffa0c03cef0 2024-09-10T20:34:55.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.801+0000 7ffa1e7fc640 10 client.54167.objecter ms_handle_connect 0x7ffa0c043e40 2024-09-10T20:34:55.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.801+0000 7ffa2d31f640 1 --2- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6808/2953083416 conn(0x7ffa0c043e40 0x7ffa0c0462e0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.805+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2463816948 <== osd.3 v2:172.21.15.176:6808/2953083416 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffa2401ce50 con 0x7ffa0c043e40 2024-09-10T20:34:55.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.805+0000 7ffa1e7fc640 10 client.54167.objecter ms_dispatch 0x7ffa28000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:34:55.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.805+0000 7ffa1e7fc640 10 client.54167.objecter _finish_command 1 = system:0 2024-09-10T20:34:55.821 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.805+0000 7ffa1e7fc640 20 client.54167.objecter put_session s=0x7ffa0c043790 osd=3 4 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.805+0000 7ffa1e7fc640 15 client.54167.objecter _session_command_op_remove 3 1 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 10 client.54167.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 15 client.54167.objecter _session_command_op_assign -1 2 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 20 client.54167.objecter _get_session s=0x7ffa0c043790 osd=3 3 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 20 client.54167.objecter put_session s=0x7ffa0c043790 osd=3 3 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 20 client.54167.objecter _get_session s=0x7ffa0c043790 osd=3 3 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 15 client.54167.objecter _session_command_op_remove -1 2 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 20 client.54167.objecter get_session s=0x7ffa0c043790 osd=3 3 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 15 client.54167.objecter _session_command_op_assign 3 2 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 20 client.54167.objecter put_session s=0x7ffa0c043790 osd=3 4 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 10 client.54167.objecter _send_command 2 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 --> v2:172.21.15.176:6808/2953083416 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ffa2813cf00 con 0x7ffa0c043e40 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa1e7fc640 1 -- 172.21.15.153:0/2463816948 <== osd.3 v2:172.21.15.176:6808/2953083416 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ffa2813cf00 con 0x7ffa0c043e40 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa1e7fc640 10 client.54167.objecter ms_dispatch 0x7ffa28000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:34:55.822 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa1e7fc640 10 client.54167.objecter _finish_command 2 = system:0 2024-09-10T20:34:55.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa1e7fc640 20 client.54167.objecter put_session s=0x7ffa0c043790 osd=3 4 2024-09-10T20:34:55.823 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa1e7fc640 15 client.54167.objecter _session_command_op_remove 3 2 2024-09-10T20:34:55.823 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:34:55.823 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:34:55.823 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:34:55.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 10 librados: watch_flush enter 2024-09-10T20:34:55.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 10 librados: watch_flush exit 2024-09-10T20:34:55.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 10 client.54167.objecter close_session for osd.3 2024-09-10T20:34:55.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6808/2953083416 conn(0x7ffa0c043e40 msgr2=0x7ffa0c0462e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 --2- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6808/2953083416 conn(0x7ffa0c043e40 0x7ffa0c0462e0 crc :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 20 client.54167.objecter put_session s=0x7ffa0c043790 osd=3 1 2024-09-10T20:34:55.824 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 20 client.54167.objecter shutdown clearing up homeless session... 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 10 client.54167.objecter successfully canceled tick 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6812/3003473398 conn(0x7ffa0c03cef0 msgr2=0x7ffa0c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 --2- 172.21.15.153:0/2463816948 >> v2:172.21.15.176:6812/3003473398 conn(0x7ffa0c03cef0 0x7ffa0c03f3b0 secure :-1 s=READY pgs=3382 cs=0 l=1 rev1=1 crypto rx=0x7ffa2000f3c0 tx=0x7ffa20003e70 comp rx=0 tx=0).stop 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 10 monclient: shutdown 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a60e0 msgr2=0x7ffa2806ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 --2- 172.21.15.153:0/2463816948 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ffa280a60e0 0x7ffa2806ed20 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400e880 tx=0x7ffa2400ed40 comp rx=0 tx=0).stop 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2e321640 1 -- 172.21.15.153:0/2463816948 reap_dead start 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 shutdown_connections 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 >> 172.21.15.153:0/2463816948 conn(0x7ffa280a9620 msgr2=0x7ffa280a71f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 shutdown_connections 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 -- 172.21.15.153:0/2463816948 wait complete. 2024-09-10T20:34:55.825 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.817+0000 7ffa2fd84640 1 librados: shutdown 2024-09-10T20:34:55.837 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-10T20:34:55.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient: get_monmap_and_config 2024-09-10T20:34:55.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient: build_initial_monmap 2024-09-10T20:34:55.951 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient: monmap: 2024-09-10T20:34:55.951 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:55.951 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.951 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:55.951098+0000 2024-09-10T20:34:55.951 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:55.951098+0000 2024-09-10T20:34:55.951 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:55.952 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.952 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:55.952 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:55.952 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:55.952 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:55.952 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:55.952 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 1 Processor -- start 2024-09-10T20:34:55.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 1 -- start start 2024-09-10T20:34:55.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient: init 2024-09-10T20:34:55.954 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient: _add_conns ranks=[2,1,4,0,3] 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1441364f0 0x7fa1441368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient(hunting): picked mon.noname-b con 0x7fa1441364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1441360a0 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fa1440a1990 0x7fa14409d520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient(hunting): picked mon.noname-c con 0x7fa1440a1990 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1440a27a0 con 0x7fa1441360a0 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fa1440a2a80 con 0x7fa1440a1990 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa1440a2d60 con 0x7fa1441364f0 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa148896640 10 monclient(hunting): authenticate will time out at 1456.090674s 2024-09-10T20:34:55.955 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1441364f0 0x7fa1441368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1441364f0 0x7fa1441368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37496/0 (socket says 172.21.15.153:37496) 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 1 -- 172.21.15.153:0/4097499698 learned_addr learned my addr 172.21.15.153:0/4097499698 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient(hunting): get_auth_request con 0x7fa1441364f0 auth_method 0 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa142d9d640 1 --2- 172.21.15.153:0/4097499698 >> v2:172.21.15.176:3300/0 conn(0x7fa1440a1990 0x7fa14409d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa142d9d640 10 monclient(hunting): get_auth_request con 0x7fa1440a1990 auth_method 0 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa142d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.956 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa142d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa142d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient(hunting): handle_auth_done global_id 54170 payload 274 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 1 -- 172.21.15.153:0/4097499698 >> v2:172.21.15.176:3300/0 conn(0x7fa1440a1990 msgr2=0x7fa14409d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 1 --2- 172.21.15.153:0/4097499698 >> v2:172.21.15.176:3300/0 conn(0x7fa1440a1990 0x7fa14409d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 1 -- 172.21.15.153:0/4097499698 >> v1:172.21.15.176:6789/0 conn(0x7fa1441360a0 legacy=0x7fa14409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 1 monclient: found mon.noname-b 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.957 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.949+0000 7fa14259c640 1 -- 172.21.15.153:0/4097499698 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa144136e10 con 0x7fa1441364f0 2024-09-10T20:34:55.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14259c640 1 -- 172.21.15.153:0/4097499698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1441364f0 msgr2=0x7fa1441368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14259c640 1 --2- 172.21.15.153:0/4097499698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1441364f0 0x7fa1441368d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa134009870 tx=0x7fa13401c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb7aa855759c3d4e server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 1 -- 172.21.15.153:0/4097499698 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1340b1070 con 0x7fa1441364f0 2024-09-10T20:34:55.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:55.961 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 10 monclient: dump: 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.962 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 1 -- 172.21.15.153:0/4097499698 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa14409dc70 con 0x7fa1441364f0 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 1 -- 172.21.15.153:0/4097499698 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1340ac840 con 0x7fa1441364f0 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa14159a640 10 monclient: dump: 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.963 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa148896640 5 monclient: authenticate success, global_id 54170 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa148896640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.953+0000 7fa148896640 4 set_mon_vals no callback set 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient: shutdown 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 -- 172.21.15.153:0/4097499698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1441364f0 msgr2=0x7fa1441368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 --2- 172.21.15.153:0/4097499698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1441364f0 0x7fa1441368d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa134009870 tx=0x7fa13401c420 comp rx=0 tx=0).stop 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 -- 172.21.15.153:0/4097499698 shutdown_connections 2024-09-10T20:34:55.964 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 --2- 172.21.15.153:0/4097499698 >> v2:172.21.15.176:3300/0 conn(0x7fa1440a1990 0x7fa14409d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 --2- 172.21.15.153:0/4097499698 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1441364f0 0x7fa1441368d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 -- 172.21.15.153:0/4097499698 >> 172.21.15.153:0/4097499698 conn(0x7fa1440a71b0 msgr2=0x7fa1440a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 -- 172.21.15.153:0/4097499698 shutdown_connections 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 -- 172.21.15.153:0/4097499698 wait complete. 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient: build_initial_monmap 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient: monmap: 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:55.965 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 librados: starting msgr at 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 librados: starting objecter 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 Processor -- start 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 -- start start 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 librados: setting wanted keys 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 librados: calling monclient init 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient: init 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:55.966 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1440a1990 0x7fa14413d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient(hunting): picked mon.noname-c con 0x7fa1440a1990 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1441360a0 0x7fa14413db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient(hunting): picked mon.noname-b con 0x7fa1441360a0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441364f0 0x7fa144071af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient(hunting): picked mon.noname-a con 0x7fa1441364f0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fa144140ff0 con 0x7fa1441364f0 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fa144140e70 con 0x7fa1441360a0 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fa144141170 con 0x7fa1440a1990 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:55.967 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa148896640 10 monclient(hunting): authenticate will time out at 1456.098300s 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1440a1990 0x7fa14413d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa142d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441364f0 0x7fa144071af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1440a1990 0x7fa14413d600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37508/0 (socket says 172.21.15.153:37508) 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 1 -- 172.21.15.153:0/3031508591 learned_addr learned my addr 172.21.15.153:0/3031508591 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient(hunting): get_auth_request con 0x7fa1440a1990 auth_method 0 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa142d9d640 10 monclient(hunting): get_auth_request con 0x7fa1441364f0 auth_method 0 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa142d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa142d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa142d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.968 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa142d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa142d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa142d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa141d9b640 1 --2- 172.21.15.153:0/3031508591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1441360a0 0x7fa14413db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa141d9b640 10 monclient(hunting): get_auth_request con 0x7fa1441360a0 auth_method 0 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa141d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa141d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa141d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient(hunting): handle_auth_done global_id 54173 payload 824 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 1 -- 172.21.15.153:0/3031508591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1441360a0 msgr2=0x7fa14413db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 1 --2- 172.21.15.153:0/3031508591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1441360a0 0x7fa14413db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 1 -- 172.21.15.153:0/3031508591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441364f0 msgr2=0x7fa144071af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 1 --2- 172.21.15.153:0/3031508591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441364f0 0x7fa144071af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:55.969 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient: _finish_hunting 0 2024-09-10T20:34:55.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 1 monclient: found mon.noname-c 2024-09-10T20:34:55.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:55.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa14259c640 1 -- 172.21.15.153:0/3031508591 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa144138090 con 0x7fa1440a1990 2024-09-10T20:34:55.970 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.957+0000 7fa142d9d640 1 --2- 172.21.15.153:0/3031508591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441364f0 0x7fa144071af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:55.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa14259c640 1 -- 172.21.15.153:0/3031508591 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1440a1990 msgr2=0x7fa14413d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:55.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa14259c640 1 --2- 172.21.15.153:0/3031508591 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1440a1990 0x7fa14413d600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa13401aad0 tx=0x7fa13401ac60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 client.?.objecter ms_handle_connect 0x7fa1440a1990 2024-09-10T20:34:55.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:55.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 1 -- 172.21.15.153:0/3031508591 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1340bb020 con 0x7fa1440a1990 2024-09-10T20:34:55.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:55.972 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 monclient: dump: 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 monclient: _finish_auth 0 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:55.973 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 1 -- 172.21.15.153:0/3031508591 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1340acdc0 con 0x7fa1440a1990 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 1 -- 172.21.15.153:0/3031508591 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1340b1040 con 0x7fa1440a1990 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 monclient: dump: 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:55.974 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa140d99640 4 set_mon_vals no callback set 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 5 monclient: authenticate success, global_id 54173 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 10 monclient: _renew_subs 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa14409e0a0 con 0x7fa1440a1990 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 10 client.54173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 10 monclient: _renew_subs 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fa144073520 con 0x7fa1440a1990 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 1 librados: init done 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.975 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 1 -- 172.21.15.153:0/3031508591 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fa1340be020 con 0x7fa1440a1990 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1441368d0 con 0x7fa1440a1990 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 1 --2- 172.21.15.153:0/3031508591 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa12403cea0 0x7fa12403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 1 -- 172.21.15.153:0/3031508591 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7fa1340e9800 con 0x7fa1440a1990 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 10 client.54173.objecter ms_dispatch 0x7fa144000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 3 client.54173.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 3 client.54173.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa132ffd640 20 client.54173.objecter dump_active .. 0 homeless 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa141d9b640 1 --2- 172.21.15.153:0/3031508591 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa12403cea0 0x7fa12403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.961+0000 7fa141d9b640 10 monclient: get_auth_request con 0x7fa12403cea0 auth_method 0 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.965+0000 7fa132ffd640 10 client.54173.objecter ms_handle_connect 0x7fa12403cea0 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.965+0000 7fa141d9b640 1 --2- 172.21.15.153:0/3031508591 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa12403cea0 0x7fa12403f360 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fa144131b10 tx=0x7fa138009040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:55.976 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.965+0000 7fa132ffd640 1 -- 172.21.15.153:0/3031508591 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fa1340c5030 con 0x7fa1440a1990 2024-09-10T20:34:55.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.965+0000 7fa132ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:55.977 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:55.965+0000 7fa132ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:56.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.121+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fa14409dc70 con 0x7fa12403cea0 2024-09-10T20:34:56.125 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa132ffd640 1 -- 172.21.15.153:0/3031508591 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fa14409dc70 con 0x7fa12403cea0 2024-09-10T20:34:56.126 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 3 to deep-scrub 2024-09-10T20:34:56.128 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 10 librados: watch_flush enter 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 10 librados: watch_flush exit 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 20 client.54173.objecter shutdown clearing up homeless session... 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 10 client.54173.objecter successfully canceled tick 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa12403cea0 msgr2=0x7fa12403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 --2- 172.21.15.153:0/3031508591 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa12403cea0 0x7fa12403f360 secure :-1 s=READY pgs=3383 cs=0 l=1 rev1=1 crypto rx=0x7fa144131b10 tx=0x7fa138009040 comp rx=0 tx=0).stop 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 10 monclient: shutdown 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1440a1990 msgr2=0x7fa14413d600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 --2- 172.21.15.153:0/3031508591 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1440a1990 0x7fa14413d600 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa13401aad0 tx=0x7fa13401ac60 comp rx=0 tx=0).stop 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 shutdown_connections 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 --2- 172.21.15.153:0/3031508591 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fa1441364f0 0x7fa144071af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 --2- 172.21.15.153:0/3031508591 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fa1441360a0 0x7fa14413db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 --2- 172.21.15.153:0/3031508591 >> v2:172.21.15.176:6812/3003473398 conn(0x7fa12403cea0 0x7fa12403f360 unknown :-1 s=CLOSED pgs=3383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.129 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 --2- 172.21.15.153:0/3031508591 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fa1440a1990 0x7fa14413d600 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 >> 172.21.15.153:0/3031508591 conn(0x7fa1440a71b0 msgr2=0x7fa1440a63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:56.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 shutdown_connections 2024-09-10T20:34:56.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 -- 172.21.15.153:0/3031508591 wait complete. 2024-09-10T20:34:56.130 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.125+0000 7fa148896640 1 librados: shutdown 2024-09-10T20:34:56.144 INFO:tasks.ceph:Scrubbing osd.4 2024-09-10T20:34:56.144 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:34:56.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient: get_monmap_and_config 2024-09-10T20:34:56.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient: build_initial_monmap 2024-09-10T20:34:56.267 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient: monmap: 2024-09-10T20:34:56.267 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:56.268 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.268 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:56.266371+0000 2024-09-10T20:34:56.268 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:56.266371+0000 2024-09-10T20:34:56.268 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:56.268 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.269 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:56.269 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:56.269 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:56.269 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:56.270 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:56.270 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.270 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 1 Processor -- start 2024-09-10T20:34:56.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 1 -- start start 2024-09-10T20:34:56.271 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient: init 2024-09-10T20:34:56.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:56.280 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient: _add_conns ranks=[2,1,3,4,0] 2024-09-10T20:34:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5040a3f50 0x7ff504132d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient(hunting): picked mon.noname-b con 0x7ff5040a3f50 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff504074500 addr v1:172.21.15.176:6789/0 2024-09-10T20:34:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff504074920 0x7ff5041367d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient(hunting): picked mon.noname-a con 0x7ff504074920 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 1 -- --> v1:172.21.15.176:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff504074d70 con 0x7ff504074500 2024-09-10T20:34:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7ff5040a09f0 con 0x7ff504074920 2024-09-10T20:34:56.281 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff5040a4330 con 0x7ff5040a3f50 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff50ac19640 10 monclient(hunting): authenticate will time out at 1456.405966s 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff504074920 0x7ff5041367d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5040a3f50 0x7ff504132d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7ff504074920 0x7ff5041367d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56318/0 (socket says 172.21.15.153:56318) 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 1 -- 172.21.15.153:0/1375228246 learned_addr learned my addr 172.21.15.153:0/1375228246 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 10 monclient(hunting): get_auth_request con 0x7ff504074920 auth_method 0 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient(hunting): get_auth_request con 0x7ff5040a3f50 auth_method 0 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.282 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient(hunting): handle_auth_done global_id 54176 payload 274 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 1 -- 172.21.15.153:0/1375228246 >> v2:172.21.15.153:3300/0 conn(0x7ff504074920 msgr2=0x7ff5041367d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 1 --2- 172.21.15.153:0/1375228246 >> v2:172.21.15.153:3300/0 conn(0x7ff504074920 0x7ff5041367d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.283 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 1 -- 172.21.15.153:0/1375228246 >> v1:172.21.15.176:6789/0 conn(0x7ff504074500 legacy=0x7ff5041360c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient: _finish_hunting 0 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 1 monclient: found mon.noname-b 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5089b5640 1 -- 172.21.15.153:0/1375228246 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5040a2dc0 con 0x7ff5040a3f50 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.265+0000 7ff5091b6640 1 --2- 172.21.15.153:0/1375228246 >> v2:172.21.15.153:3300/0 conn(0x7ff504074920 0x7ff5041367d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5089b5640 1 -- 172.21.15.153:0/1375228246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5040a3f50 msgr2=0x7ff504132d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5089b5640 1 --2- 172.21.15.153:0/1375228246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5040a3f50 0x7ff504132d80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40097c0 tx=0x7ff4f401c150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e07ad8314027498f server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 1 -- 172.21.15.153:0/1375228246 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f40b0070 con 0x7ff5040a3f50 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 10 monclient: dump: 2024-09-10T20:34:56.284 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 10 monclient: _finish_auth 0 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 10 monclient: _check_auth_tickets 2024-09-10T20:34:56.285 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 1 -- 172.21.15.153:0/1375228246 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f40ab600 con 0x7ff5040a3f50 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 5 monclient: authenticate success, global_id 54176 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 1 -- 172.21.15.153:0/1375228246 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f40ab9d0 con 0x7ff5040a3f50 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 4 set_mon_vals no callback set 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff5037fe640 10 monclient: dump: 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.286 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 10 monclient: shutdown 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 1 -- 172.21.15.153:0/1375228246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5040a3f50 msgr2=0x7ff504132d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 1 --2- 172.21.15.153:0/1375228246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5040a3f50 0x7ff504132d80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7ff4f40097c0 tx=0x7ff4f401c150 comp rx=0 tx=0).stop 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 1 -- 172.21.15.153:0/1375228246 shutdown_connections 2024-09-10T20:34:56.287 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 1 --2- 172.21.15.153:0/1375228246 >> v2:172.21.15.153:3300/0 conn(0x7ff504074920 0x7ff5041367d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 1 --2- 172.21.15.153:0/1375228246 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff5040a3f50 0x7ff504132d80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 1 -- 172.21.15.153:0/1375228246 >> 172.21.15.153:0/1375228246 conn(0x7ff5040a7900 msgr2=0x7ff5040a3840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 1 -- 172.21.15.153:0/1375228246 shutdown_connections 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.269+0000 7ff50ac19640 1 -- 172.21.15.153:0/1375228246 wait complete. 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient: build_initial_monmap 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient: monmap: 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:56.288 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 librados: starting msgr at 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 librados: starting objecter 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 Processor -- start 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 -- start start 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 librados: setting wanted keys 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 librados: calling monclient init 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient: init 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:56.289 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff504074500 0x7ff50413d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient(hunting): picked mon.noname-b con 0x7ff504074500 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff504074920 0x7ff5041ce340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient(hunting): picked mon.noname-c con 0x7ff504074920 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5040a3f50 0x7ff5041ce880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient(hunting): picked mon.noname-a con 0x7ff5040a3f50 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7ff504136e90 con 0x7ff5040a3f50 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7ff504136d10 con 0x7ff504074500 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.290 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7ff504140c80 con 0x7ff504074920 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient(hunting): authenticate will time out at 1456.412944s 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff504074920 0x7ff5041ce340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff504074920 0x7ff5041ce340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37530/0 (socket says 172.21.15.153:37530) 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 -- 172.21.15.153:0/1286089031 learned_addr learned my addr 172.21.15.153:0/1286089031 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient(hunting): get_auth_request con 0x7ff504074920 auth_method 0 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5091b6640 1 --2- 172.21.15.153:0/1286089031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5040a3f50 0x7ff5041ce880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5091b6640 10 monclient(hunting): get_auth_request con 0x7ff5040a3f50 auth_method 0 2024-09-10T20:34:56.291 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5091b6640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5091b6640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5091b6640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5091b6640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5091b6640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5091b6640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient(hunting): handle_auth_done global_id 54179 payload 824 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 -- 172.21.15.153:0/1286089031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff504074500 msgr2=0x7ff50413d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5089b5640 1 --2- 172.21.15.153:0/1286089031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff504074500 0x7ff50413d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.292 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 --2- 172.21.15.153:0/1286089031 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7ff504074500 0x7ff50413d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 -- 172.21.15.153:0/1286089031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5040a3f50 msgr2=0x7ff5041ce880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 --2- 172.21.15.153:0/1286089031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5040a3f50 0x7ff5041ce880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient: _finish_hunting 0 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 monclient: found mon.noname-c 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 -- 172.21.15.153:0/1286089031 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff50409e0a0 con 0x7ff504074920 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5091b6640 1 --2- 172.21.15.153:0/1286089031 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7ff5040a3f50 0x7ff5041ce880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 -- 172.21.15.153:0/1286089031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff504074920 msgr2=0x7ff5041ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff503fff640 1 --2- 172.21.15.153:0/1286089031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff504074920 0x7ff5041ce340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800c480 tx=0x7ff4f800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 client.?.objecter ms_handle_connect 0x7ff504074920 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:56.293 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 1 -- 172.21.15.153:0/1286089031 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8012a40 con 0x7ff504074920 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 monclient: dump: 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.294 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 monclient: _finish_auth 0 2024-09-10T20:34:56.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 monclient: _check_auth_tickets 2024-09-10T20:34:56.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:56.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 1 -- 172.21.15.153:0/1286089031 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4f8017070 con 0x7ff504074920 2024-09-10T20:34:56.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 5 monclient: authenticate success, global_id 54179 2024-09-10T20:34:56.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient: _renew_subs 2024-09-10T20:34:56.295 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5040732b0 con 0x7ff504074920 2024-09-10T20:34:56.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:56.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 1 -- 172.21.15.153:0/1286089031 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4f8012e40 con 0x7ff504074920 2024-09-10T20:34:56.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:56.296 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff5017fa640 10 monclient: dump: 2024-09-10T20:34:56.296 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.296 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.296 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff502ffd640 4 set_mon_vals no callback set 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 client.54179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient: _renew_subs 2024-09-10T20:34:56.297 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff504137950 con 0x7ff504074920 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.273+0000 7ff50ac19640 1 librados: init done 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff50ac19640 10 client.54179.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff50ac19640 15 client.54179.objecter _session_command_op_assign -1 1 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff50ac19640 20 client.54179.objecter _get_session osd=-1 returning homeless 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff50ac19640 10 client.54179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff50ac19640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff504132d80 con 0x7ff504074920 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 1 -- 172.21.15.153:0/1286089031 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7ff4f801c460 con 0x7ff504074920 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 1 --2- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff4f003cea0 0x7ff4f003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 1 -- 172.21.15.153:0/1286089031 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7ff4f804f390 con 0x7ff504074920 2024-09-10T20:34:56.298 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 10 client.54179.objecter ms_dispatch 0x7ff504000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 3 client.54179.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 3 client.54179.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 10 client.54179.objecter checking command 1 2024-09-10T20:34:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 1 --2- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6804/4030285519 conn(0x7ff4f0043d70 0x7ff4f0046210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 20 client.54179.objecter _get_session s=0x7ff4f00436c0 osd=4 3 2024-09-10T20:34:56.299 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 20 client.54179.objecter put_session s=0x7ff4f00436c0 osd=4 3 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 15 client.54179.objecter _session_command_op_remove -1 1 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 20 client.54179.objecter _get_session s=0x7ff4f00436c0 osd=4 3 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 20 client.54179.objecter get_session s=0x7ff4f00436c0 osd=4 3 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 15 client.54179.objecter _session_command_op_assign 4 1 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 20 client.54179.objecter put_session s=0x7ff4f00436c0 osd=4 4 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 10 client.54179.objecter _send_command 1 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 1 -- 172.21.15.153:0/1286089031 --> v2:172.21.15.176:6804/4030285519 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff4f801ae50 con 0x7ff4f0043d70 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 20 client.54179.objecter dump_active .. 0 homeless 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 1 -- 172.21.15.153:0/1286089031 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=317) ==== 24+0+0 (secure 0 0 0) 0x7ff4f8056050 con 0x7ff504074920 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5017fa640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5089b5640 1 --2- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff4f003cea0 0x7ff4f003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5089b5640 10 monclient: get_auth_request con 0x7ff4f003cea0 auth_method 0 2024-09-10T20:34:56.300 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5091b6640 1 --2- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6804/4030285519 conn(0x7ff4f0043d70 0x7ff4f0046210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.301 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.277+0000 7ff5091b6640 10 monclient: get_auth_request con 0x7ff4f0043d70 auth_method 0 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.281+0000 7ff5089b5640 1 --2- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff4f003cea0 0x7ff4f003f360 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7ff5040a4660 tx=0x7ff4f4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.281+0000 7ff5017fa640 10 client.54179.objecter ms_handle_connect 0x7ff4f003cea0 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.281+0000 7ff5091b6640 1 --2- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6804/4030285519 conn(0x7ff4f0043d70 0x7ff4f0046210 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.281+0000 7ff5017fa640 10 client.54179.objecter ms_handle_connect 0x7ff4f0043d70 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.285+0000 7ff5017fa640 1 -- 172.21.15.153:0/1286089031 <== osd.4 v2:172.21.15.176:6804/4030285519 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff4f801ae50 con 0x7ff4f0043d70 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.285+0000 7ff5017fa640 10 client.54179.objecter ms_dispatch 0x7ff504000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.285+0000 7ff5017fa640 10 client.54179.objecter _finish_command 1 = system:0 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.285+0000 7ff5017fa640 20 client.54179.objecter put_session s=0x7ff4f00436c0 osd=4 4 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.285+0000 7ff5017fa640 15 client.54179.objecter _session_command_op_remove 4 1 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 10 client.54179.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-10T20:34:56.302 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 15 client.54179.objecter _session_command_op_assign -1 2 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 20 client.54179.objecter _get_session s=0x7ff4f00436c0 osd=4 3 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 20 client.54179.objecter put_session s=0x7ff4f00436c0 osd=4 3 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 20 client.54179.objecter _get_session s=0x7ff4f00436c0 osd=4 3 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 15 client.54179.objecter _session_command_op_remove -1 2 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 20 client.54179.objecter get_session s=0x7ff4f00436c0 osd=4 3 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 15 client.54179.objecter _session_command_op_assign 4 2 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 20 client.54179.objecter put_session s=0x7ff4f00436c0 osd=4 4 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 10 client.54179.objecter _send_command 2 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 --> v2:172.21.15.176:6804/4030285519 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff5040a0550 con 0x7ff4f0043d70 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff5017fa640 1 -- 172.21.15.153:0/1286089031 <== osd.4 v2:172.21.15.176:6804/4030285519 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff5040a0550 con 0x7ff4f0043d70 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff5017fa640 10 client.54179.objecter ms_dispatch 0x7ff504000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff5017fa640 10 client.54179.objecter _finish_command 2 = system:0 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff5017fa640 20 client.54179.objecter put_session s=0x7ff4f00436c0 osd=4 4 2024-09-10T20:34:56.303 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff5017fa640 15 client.54179.objecter _session_command_op_remove 4 2 2024-09-10T20:34:56.304 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:34:56.304 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:34:56.304 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:34:56.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 10 librados: watch_flush enter 2024-09-10T20:34:56.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 10 librados: watch_flush exit 2024-09-10T20:34:56.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 10 client.54179.objecter close_session for osd.4 2024-09-10T20:34:56.306 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6804/4030285519 conn(0x7ff4f0043d70 msgr2=0x7ff4f0046210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 --2- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6804/4030285519 conn(0x7ff4f0043d70 0x7ff4f0046210 crc :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 20 client.54179.objecter put_session s=0x7ff4f00436c0 osd=4 1 2024-09-10T20:34:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 20 client.54179.objecter shutdown clearing up homeless session... 2024-09-10T20:34:56.307 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 10 client.54179.objecter successfully canceled tick 2024-09-10T20:34:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff4f003cea0 msgr2=0x7ff4f003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 --2- 172.21.15.153:0/1286089031 >> v2:172.21.15.176:6812/3003473398 conn(0x7ff4f003cea0 0x7ff4f003f360 secure :-1 s=READY pgs=3384 cs=0 l=1 rev1=1 crypto rx=0x7ff5040a4660 tx=0x7ff4f4008040 comp rx=0 tx=0).stop 2024-09-10T20:34:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 10 monclient: shutdown 2024-09-10T20:34:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:56.308 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff504074920 msgr2=0x7ff5041ce340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 --2- 172.21.15.153:0/1286089031 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7ff504074920 0x7ff5041ce340 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7ff4f800c480 tx=0x7ff4f800c940 comp rx=0 tx=0).stop 2024-09-10T20:34:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff5091b6640 1 -- 172.21.15.153:0/1286089031 reap_dead start 2024-09-10T20:34:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 shutdown_connections 2024-09-10T20:34:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 >> 172.21.15.153:0/1286089031 conn(0x7ff5040a7900 msgr2=0x7ff504130ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 shutdown_connections 2024-09-10T20:34:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 -- 172.21.15.153:0/1286089031 wait complete. 2024-09-10T20:34:56.309 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.297+0000 7ff50ac19640 1 librados: shutdown 2024-09-10T20:34:56.316 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-10T20:34:56.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient: get_monmap_and_config 2024-09-10T20:34:56.426 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient: build_initial_monmap 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient: monmap: 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:56.426480+0000 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:56.426480+0000 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:56.427 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 1 Processor -- start 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 1 -- start start 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient: init 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient: _add_conns ranks=[3,4,0,2,1] 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8641364f0 0x7fe8641368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient(hunting): picked mon.noname-a con 0x7fe8641364f0 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fe8641360a0 0x7fe86409ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient(hunting): picked mon.noname-c con 0x7fe8641360a0 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe8640a1990 addr v1:172.21.15.153:6789/0 2024-09-10T20:34:56.430 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8640a27a0 con 0x7fe8640a1990 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fe8640a2a80 con 0x7fe8641364f0 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fe8640a2d60 con 0x7fe8641360a0 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe86a0f2640 10 monclient(hunting): authenticate will time out at 1456.566143s 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fe8641364f0 0x7fe8641368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe863fff640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fe8640a1990 0x7fe86409d350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:60740/0 (socket says 172.21.15.153:60740) 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe863fff640 1 -- 172.21.15.153:0/2631716999 learned_addr learned my addr 172.21.15.153:0/2631716999 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient(hunting): get_auth_request con 0x7fe8641364f0 auth_method 0 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.431 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe862ffd640 1 --2- 172.21.15.153:0/2631716999 >> v2:172.21.15.176:3300/0 conn(0x7fe8641360a0 0x7fe86409ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient(hunting): handle_auth_done global_id 34440 payload 274 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 1 -- 172.21.15.153:0/2631716999 >> v2:172.21.15.176:3300/0 conn(0x7fe8641360a0 msgr2=0x7fe86409ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 1 --2- 172.21.15.153:0/2631716999 >> v2:172.21.15.176:3300/0 conn(0x7fe8641360a0 0x7fe86409ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 1 -- 172.21.15.153:0/2631716999 >> v1:172.21.15.153:6789/0 conn(0x7fe8640a1990 legacy=0x7fe86409d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient: _finish_hunting 0 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 1 monclient: found mon.noname-a 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:56.432 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe8637fe640 1 -- 172.21.15.153:0/2631716999 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe864136e10 con 0x7fe8641364f0 2024-09-10T20:34:56.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe862ffd640 10 monclient: get_auth_request con 0x7fe8641360a0 auth_method 0 2024-09-10T20:34:56.433 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.425+0000 7fe862ffd640 1 --2- 172.21.15.153:0/2631716999 >> v2:172.21.15.176:3300/0 conn(0x7fe8641360a0 0x7fe86409ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:56.434 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8637fe640 1 -- 172.21.15.153:0/2631716999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8641364f0 msgr2=0x7fe8641368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8637fe640 1 --2- 172.21.15.153:0/2631716999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8641364f0 0x7fe8641368d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe8640a13d0 tx=0x7fe84c01c3d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c79947d522141fcc server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 1 -- 172.21.15.153:0/2631716999 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c0b0070 con 0x7fe8641364f0 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 10 monclient: dump: 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.435 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.436 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 10 monclient: _finish_auth 0 2024-09-10T20:34:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 10 monclient: _check_auth_tickets 2024-09-10T20:34:56.436 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 1 -- 172.21.15.153:0/2631716999 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe84c0ab600 con 0x7fe8641364f0 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 5 monclient: authenticate success, global_id 34440 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 1 -- 172.21.15.153:0/2631716999 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c0ab9d0 con 0x7fe8641364f0 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 4 set_mon_vals no callback set 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe8627fc640 10 monclient: dump: 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.437 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 10 monclient: shutdown 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 1 -- 172.21.15.153:0/2631716999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8641364f0 msgr2=0x7fe8641368d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 1 --2- 172.21.15.153:0/2631716999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8641364f0 0x7fe8641368d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe8640a13d0 tx=0x7fe84c01c3d0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.438 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 1 -- 172.21.15.153:0/2631716999 shutdown_connections 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 1 --2- 172.21.15.153:0/2631716999 >> v2:172.21.15.176:3300/0 conn(0x7fe8641360a0 0x7fe86409ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 1 --2- 172.21.15.153:0/2631716999 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8641364f0 0x7fe8641368d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 1 -- 172.21.15.153:0/2631716999 >> 172.21.15.153:0/2631716999 conn(0x7fe8640a71b0 msgr2=0x7fe8640a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 1 -- 172.21.15.153:0/2631716999 shutdown_connections 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.429+0000 7fe86a0f2640 1 -- 172.21.15.153:0/2631716999 wait complete. 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient: build_initial_monmap 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient: monmap: 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:56.439 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 librados: starting msgr at 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 librados: starting objecter 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 Processor -- start 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 -- start start 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 librados: setting wanted keys 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 librados: calling monclient init 2024-09-10T20:34:56.440 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient: init 2024-09-10T20:34:56.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:56.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-10T20:34:56.444 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8640a1990 0x7fe8641370d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient(hunting): picked mon.noname-a con 0x7fe8640a1990 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8641360a0 0x7fe864137610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient(hunting): picked mon.noname-c con 0x7fe8641360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8641364f0 0x7fe864137b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient(hunting): picked mon.noname-b con 0x7fe8641364f0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fe864140e70 con 0x7fe8640a1990 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fe8640a08a0 con 0x7fe8641364f0 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fe864140ff0 con 0x7fe8641360a0 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:56.445 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe86a0f2640 10 monclient(hunting): authenticate will time out at 1456.573659s 2024-09-10T20:34:56.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8640a1990 0x7fe8641370d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe862ffd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8641360a0 0x7fe864137610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8640a1990 0x7fe8641370d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56332/0 (socket says 172.21.15.153:56332) 2024-09-10T20:34:56.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe862ffd640 10 monclient(hunting): get_auth_request con 0x7fe8641360a0 auth_method 0 2024-09-10T20:34:56.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 1 -- 172.21.15.153:0/3626509270 learned_addr learned my addr 172.21.15.153:0/3626509270 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:56.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe862ffd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.446 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe862ffd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient(hunting): get_auth_request con 0x7fe8640a1990 auth_method 0 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe862ffd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe862ffd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe862ffd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.447 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe863fff640 1 --2- 172.21.15.153:0/3626509270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8641364f0 0x7fe864137b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient(hunting): handle_auth_done global_id 34443 payload 824 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 1 -- 172.21.15.153:0/3626509270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8641360a0 msgr2=0x7fe864137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 1 --2- 172.21.15.153:0/3626509270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8641360a0 0x7fe864137610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 1 -- 172.21.15.153:0/3626509270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8641364f0 msgr2=0x7fe864137b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 1 --2- 172.21.15.153:0/3626509270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8641364f0 0x7fe864137b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient: _finish_hunting 0 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 1 monclient: found mon.noname-a 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe8637fe640 1 -- 172.21.15.153:0/3626509270 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe86413aa60 con 0x7fe8640a1990 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe863fff640 10 monclient: get_auth_request con 0x7fe8641364f0 auth_method 0 2024-09-10T20:34:56.448 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.433+0000 7fe863fff640 1 --2- 172.21.15.153:0/3626509270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8641364f0 0x7fe864137b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe8637fe640 1 -- 172.21.15.153:0/3626509270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8640a1990 msgr2=0x7fe8641370d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe8637fe640 1 --2- 172.21.15.153:0/3626509270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8640a1990 0x7fe8641370d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe84c01aa80 tx=0x7fe84c019170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 client.?.objecter ms_handle_connect 0x7fe8640a1990 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 1 -- 172.21.15.153:0/3626509270 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c0b0040 con 0x7fe8640a1990 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 monclient: dump: 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.449 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 monclient: _finish_auth 0 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 monclient: _check_auth_tickets 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 1 -- 172.21.15.153:0/3626509270 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe84c0b9070 con 0x7fe8640a1990 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 1 -- 172.21.15.153:0/3626509270 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe84c0aa9f0 con 0x7fe8640a1990 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.450 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 monclient: dump: 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 5 monclient: authenticate success, global_id 34443 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 10 monclient: _renew_subs 2024-09-10T20:34:56.451 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe8617fa640 4 set_mon_vals no callback set 2024-09-10T20:34:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe86413ad40 con 0x7fe8640a1990 2024-09-10T20:34:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 10 client.34443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 10 monclient: _renew_subs 2024-09-10T20:34:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe86409e0a0 con 0x7fe8640a1990 2024-09-10T20:34:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 1 librados: init done 2024-09-10T20:34:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:56.452 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe86413dc40 con 0x7fe8640a1990 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 1 -- 172.21.15.153:0/3626509270 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fe84c0ab4c0 con 0x7fe8640a1990 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 1 --2- 172.21.15.153:0/3626509270 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe84403cea0 0x7fe84403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 1 -- 172.21.15.153:0/3626509270 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7fe84c0e9490 con 0x7fe8640a1990 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 10 client.34443.objecter ms_dispatch 0x7fe864000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 3 client.34443.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 3 client.34443.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe85bfff640 20 client.34443.objecter dump_active .. 0 homeless 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe862ffd640 1 --2- 172.21.15.153:0/3626509270 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe84403cea0 0x7fe84403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.437+0000 7fe862ffd640 10 monclient: get_auth_request con 0x7fe84403cea0 auth_method 0 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.441+0000 7fe862ffd640 1 --2- 172.21.15.153:0/3626509270 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe84403cea0 0x7fe84403f360 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7fe86413e180 tx=0x7fe854008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.453 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.441+0000 7fe85bfff640 10 client.34443.objecter ms_handle_connect 0x7fe84403cea0 2024-09-10T20:34:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.441+0000 7fe85bfff640 1 -- 172.21.15.153:0/3626509270 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fe84c016050 con 0x7fe8640a1990 2024-09-10T20:34:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.441+0000 7fe85bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:56.454 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.441+0000 7fe85bfff640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:56.612 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.609+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7fe8641389e0 con 0x7fe84403cea0 2024-09-10T20:34:56.615 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.609+0000 7fe85bfff640 1 -- 172.21.15.153:0/3626509270 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe8641389e0 con 0x7fe84403cea0 2024-09-10T20:34:56.615 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 4 to deep-scrub 2024-09-10T20:34:56.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 10 librados: watch_flush enter 2024-09-10T20:34:56.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 10 librados: watch_flush exit 2024-09-10T20:34:56.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 20 client.34443.objecter shutdown clearing up homeless session... 2024-09-10T20:34:56.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 10 client.34443.objecter successfully canceled tick 2024-09-10T20:34:56.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe84403cea0 msgr2=0x7fe84403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 --2- 172.21.15.153:0/3626509270 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe84403cea0 0x7fe84403f360 secure :-1 s=READY pgs=3385 cs=0 l=1 rev1=1 crypto rx=0x7fe86413e180 tx=0x7fe854008040 comp rx=0 tx=0).stop 2024-09-10T20:34:56.616 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 10 monclient: shutdown 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8640a1990 msgr2=0x7fe8641370d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 --2- 172.21.15.153:0/3626509270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8640a1990 0x7fe8641370d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fe84c01aa80 tx=0x7fe84c019170 comp rx=0 tx=0).stop 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 shutdown_connections 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 --2- 172.21.15.153:0/3626509270 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fe8641364f0 0x7fe864137b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 --2- 172.21.15.153:0/3626509270 >> v2:172.21.15.176:6812/3003473398 conn(0x7fe84403cea0 0x7fe84403f360 unknown :-1 s=CLOSED pgs=3385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 --2- 172.21.15.153:0/3626509270 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fe8641360a0 0x7fe864137610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 --2- 172.21.15.153:0/3626509270 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fe8640a1990 0x7fe8641370d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 >> 172.21.15.153:0/3626509270 conn(0x7fe8640a71b0 msgr2=0x7fe8640a61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 shutdown_connections 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 -- 172.21.15.153:0/3626509270 wait complete. 2024-09-10T20:34:56.617 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.613+0000 7fe86a0f2640 1 librados: shutdown 2024-09-10T20:34:56.631 INFO:tasks.ceph:Scrubbing osd.5 2024-09-10T20:34:56.631 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-10T20:34:56.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient: get_monmap_and_config 2024-09-10T20:34:56.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient: build_initial_monmap 2024-09-10T20:34:56.754 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient: monmap: 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:56.754509+0000 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:56.754509+0000 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:56.755 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 1 Processor -- start 2024-09-10T20:34:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 1 -- start start 2024-09-10T20:34:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient: init 2024-09-10T20:34:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient: _add_conns ranks=[3,2,4,1,0] 2024-09-10T20:34:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f69980a5ab0 0x7f69980a5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient(hunting): picked mon.noname-a con 0x7f69980a5ab0 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:56.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69980a63d0 0x7f6998138870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient(hunting): picked mon.noname-b con 0x7f69980a63d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:56.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f6998138db0 0x7f69981391d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient(hunting): picked mon.noname-c con 0x7f6998138db0 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:56.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.759 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f69980a6b00 con 0x7f69980a5ab0 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f69980a2650 con 0x7f6998138db0 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f69980a5210 con 0x7f69980a63d0 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699f33f640 10 monclient(hunting): authenticate will time out at 1456.894164s 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69980a63d0 0x7f6998138870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69980a63d0 0x7f6998138870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37556/0 (socket says 172.21.15.153:37556) 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 1 -- 172.21.15.153:0/323953328 learned_addr learned my addr 172.21.15.153:0/323953328 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d0db640 1 --2- 172.21.15.153:0/323953328 >> v2:172.21.15.153:3300/0 conn(0x7f69980a5ab0 0x7f69980a5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient(hunting): get_auth_request con 0x7f69980a63d0 auth_method 0 2024-09-10T20:34:56.760 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d0db640 10 monclient(hunting): get_auth_request con 0x7f69980a5ab0 auth_method 0 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d0db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d0db640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d0db640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d0db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d0db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d0db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d8dc640 1 --2- 172.21.15.153:0/323953328 >> v2:172.21.15.176:3300/0 conn(0x7f6998138db0 0x7f69981391d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d8dc640 10 monclient(hunting): get_auth_request con 0x7f6998138db0 auth_method 0 2024-09-10T20:34:56.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d8dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d8dc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d8dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient(hunting): handle_auth_done global_id 54185 payload 274 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 1 -- 172.21.15.153:0/323953328 >> v2:172.21.15.176:3300/0 conn(0x7f6998138db0 msgr2=0x7f69981391d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 1 --2- 172.21.15.153:0/323953328 >> v2:172.21.15.176:3300/0 conn(0x7f6998138db0 0x7f69981391d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 1 -- 172.21.15.153:0/323953328 >> v2:172.21.15.153:3300/0 conn(0x7f69980a5ab0 msgr2=0x7f69980a5e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 1 --2- 172.21.15.153:0/323953328 >> v2:172.21.15.153:3300/0 conn(0x7f69980a5ab0 0x7f69980a5e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient: _finish_hunting 0 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 1 monclient: found mon.noname-b 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699c8da640 1 -- 172.21.15.153:0/323953328 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69980a2eb0 con 0x7f69980a63d0 2024-09-10T20:34:56.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.753+0000 7f699d0db640 1 --2- 172.21.15.153:0/323953328 >> v2:172.21.15.153:3300/0 conn(0x7f69980a5ab0 0x7f69980a5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:56.768 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699c8da640 1 -- 172.21.15.153:0/323953328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69980a63d0 msgr2=0x7f6998138870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699c8da640 1 --2- 172.21.15.153:0/323953328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69980a63d0 0x7f6998138870 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6998070210 tx=0x7f698801c3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f71d688dfb04b2a9 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 1 -- 172.21.15.153:0/323953328 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69880b0070 con 0x7f69980a63d0 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 10 monclient: dump: 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.769 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 10 monclient: _finish_auth 0 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 10 monclient: _check_auth_tickets 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 1 -- 172.21.15.153:0/323953328 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69880ab600 con 0x7f69980a63d0 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 1 -- 172.21.15.153:0/323953328 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69880ab9d0 con 0x7f69980a63d0 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 5 monclient: authenticate success, global_id 54185 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 4 set_mon_vals no callback set 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:56.770 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f698ffff640 10 monclient: dump: 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 10 monclient: shutdown 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 1 -- 172.21.15.153:0/323953328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69980a63d0 msgr2=0x7f6998138870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 1 --2- 172.21.15.153:0/323953328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69980a63d0 0x7f6998138870 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6998070210 tx=0x7f698801c3e0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.771 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 1 -- 172.21.15.153:0/323953328 shutdown_connections 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 1 --2- 172.21.15.153:0/323953328 >> v2:172.21.15.176:3300/0 conn(0x7f6998138db0 0x7f69981391d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 1 --2- 172.21.15.153:0/323953328 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f69980a63d0 0x7f6998138870 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 1 --2- 172.21.15.153:0/323953328 >> v2:172.21.15.153:3300/0 conn(0x7f69980a5ab0 0x7f69980a5e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 1 -- 172.21.15.153:0/323953328 >> 172.21.15.153:0/323953328 conn(0x7f69980ab920 msgr2=0x7f69980a7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 1 -- 172.21.15.153:0/323953328 shutdown_connections 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.757+0000 7f699f33f640 1 -- 172.21.15.153:0/323953328 wait complete. 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient: build_initial_monmap 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient: monmap: 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:56.772 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 librados: starting msgr at 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 librados: starting objecter 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 Processor -- start 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 -- start start 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 librados: setting wanted keys 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 librados: calling monclient init 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient: init 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:34:56.773 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69980a5ab0 0x7f699813d000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient(hunting): picked mon.noname-a con 0x7f69980a5ab0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69980a63d0 0x7f699813d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient(hunting): picked mon.noname-b con 0x7f69980a63d0 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6998138db0 0x7f699813da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient(hunting): picked mon.noname-c con 0x7f6998138db0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f69980a8850 con 0x7f69980a5ab0 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f69980a86d0 con 0x7f69980a63d0 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f69980a38b0 con 0x7f6998138db0 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:56.774 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699f33f640 10 monclient(hunting): authenticate will time out at 1456.901224s 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6998138db0 0x7f699813da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6998138db0 0x7f699813da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37562/0 (socket says 172.21.15.153:37562) 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 -- 172.21.15.153:0/431899275 learned_addr learned my addr 172.21.15.153:0/431899275 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient(hunting): get_auth_request con 0x7f6998138db0 auth_method 0 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d0db640 1 --2- 172.21.15.153:0/431899275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69980a5ab0 0x7f699813d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d0db640 10 monclient(hunting): get_auth_request con 0x7f69980a5ab0 auth_method 0 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d0db640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d0db640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d0db640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.775 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d0db640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d0db640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d0db640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699c8da640 1 --2- 172.21.15.153:0/431899275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69980a63d0 0x7f699813d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699c8da640 10 monclient(hunting): get_auth_request con 0x7f69980a63d0 auth_method 0 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699c8da640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699c8da640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699c8da640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient(hunting): handle_auth_done global_id 54188 payload 824 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 -- 172.21.15.153:0/431899275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69980a63d0 msgr2=0x7f699813d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 --2- 172.21.15.153:0/431899275 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f69980a63d0 0x7f699813d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 -- 172.21.15.153:0/431899275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69980a5ab0 msgr2=0x7f699813d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.776 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 --2- 172.21.15.153:0/431899275 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f69980a5ab0 0x7f699813d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient: _finish_hunting 0 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 monclient: found mon.noname-c 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 -- 172.21.15.153:0/431899275 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69980a1640 con 0x7f6998138db0 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 -- 172.21.15.153:0/431899275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6998138db0 msgr2=0x7f699813da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f699d8dc640 1 --2- 172.21.15.153:0/431899275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6998138db0 0x7f699813da80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6990007e70 tx=0x7f699000e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f698dffb640 10 client.?.objecter ms_handle_connect 0x7f6998138db0 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.761+0000 7f698dffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 1 -- 172.21.15.153:0/431899275 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990014620 con 0x7f6998138db0 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: dump: 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.777 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: _finish_auth 0 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: _check_auth_tickets 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 1 -- 172.21.15.153:0/431899275 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6990019070 con 0x7f6998138db0 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 1 -- 172.21.15.153:0/431899275 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6990014a20 con 0x7f6998138db0 2024-09-10T20:34:56.778 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: dump: 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.779 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698f7fe640 4 set_mon_vals no callback set 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 5 monclient: authenticate success, global_id 54188 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 10 monclient: _renew_subs 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6998123a40 con 0x7f6998138db0 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 10 client.54188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 10 monclient: _renew_subs 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69980a6fd0 con 0x7f6998138db0 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 1 librados: init done 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 10 client.54188.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 15 client.54188.objecter _session_command_op_assign -1 1 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 20 client.54188.objecter _get_session osd=-1 returning homeless 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 10 client.54188.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:56.780 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6998138870 con 0x7f6998138db0 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 1 -- 172.21.15.153:0/431899275 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f6990014bc0 con 0x7f6998138db0 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 1 --2- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6812/3003473398 conn(0x7f697c03cef0 0x7f697c03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 1 -- 172.21.15.153:0/431899275 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f699004fdd0 con 0x7f6998138db0 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 client.54188.objecter ms_dispatch 0x7f6998000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 3 client.54188.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 3 client.54188.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 client.54188.objecter checking command 1 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 1 --2- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6800/2372911183 conn(0x7f697c043e40 0x7f697c0462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 20 client.54188.objecter _get_session s=0x7f697c043790 osd=5 3 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 20 client.54188.objecter put_session s=0x7f697c043790 osd=5 3 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 15 client.54188.objecter _session_command_op_remove -1 1 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 20 client.54188.objecter _get_session s=0x7f697c043790 osd=5 3 2024-09-10T20:34:56.781 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 20 client.54188.objecter get_session s=0x7f697c043790 osd=5 3 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 15 client.54188.objecter _session_command_op_assign 5 1 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 20 client.54188.objecter put_session s=0x7f697c043790 osd=5 4 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 client.54188.objecter _send_command 1 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 1 -- 172.21.15.153:0/431899275 --> v2:172.21.15.176:6800/2372911183 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f699000be50 con 0x7f697c043e40 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 20 client.54188.objecter dump_active .. 0 homeless 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 1 -- 172.21.15.153:0/431899275 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_get_version_reply(handle=1 version=317) ==== 24+0+0 (secure 0 0 0) 0x7f6990054030 con 0x7f6998138db0 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f698dffb640 10 monclient: handle_get_version_reply finishing 1 version 317 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699d0db640 1 --2- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6812/3003473398 conn(0x7f697c03cef0 0x7f697c03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699d0db640 10 monclient: get_auth_request con 0x7f697c03cef0 auth_method 0 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699c8da640 1 --2- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6800/2372911183 conn(0x7f697c043e40 0x7f697c0462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.782 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.765+0000 7f699c8da640 10 monclient: get_auth_request con 0x7f697c043e40 auth_method 0 2024-09-10T20:34:56.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.769+0000 7f699d0db640 1 --2- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6812/3003473398 conn(0x7f697c03cef0 0x7f697c03f3b0 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f6994009580 tx=0x7f6994009190 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.769+0000 7f698dffb640 10 client.54188.objecter ms_handle_connect 0x7f697c03cef0 2024-09-10T20:34:56.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.769+0000 7f698dffb640 10 client.54188.objecter ms_handle_connect 0x7f697c043e40 2024-09-10T20:34:56.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.769+0000 7f699c8da640 1 --2- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6800/2372911183 conn(0x7f697c043e40 0x7f697c0462e0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.773+0000 7f698dffb640 1 -- 172.21.15.153:0/431899275 <== osd.5 v2:172.21.15.176:6800/2372911183 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f699000be50 con 0x7f697c043e40 2024-09-10T20:34:56.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.773+0000 7f698dffb640 10 client.54188.objecter ms_dispatch 0x7f6998000ba0 command_reply(tid 1: 0 ) 2024-09-10T20:34:56.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.773+0000 7f698dffb640 10 client.54188.objecter _finish_command 1 = system:0 2024-09-10T20:34:56.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.773+0000 7f698dffb640 20 client.54188.objecter put_session s=0x7f697c043790 osd=5 4 2024-09-10T20:34:56.783 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.773+0000 7f698dffb640 15 client.54188.objecter _session_command_op_remove 5 1 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 10 client.54188.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 15 client.54188.objecter _session_command_op_assign -1 2 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 20 client.54188.objecter _get_session s=0x7f697c043790 osd=5 3 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 20 client.54188.objecter put_session s=0x7f697c043790 osd=5 3 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 20 client.54188.objecter _get_session s=0x7f697c043790 osd=5 3 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 15 client.54188.objecter _session_command_op_remove -1 2 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 20 client.54188.objecter get_session s=0x7f697c043790 osd=5 3 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 15 client.54188.objecter _session_command_op_assign 5 2 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 20 client.54188.objecter put_session s=0x7f697c043790 osd=5 4 2024-09-10T20:34:56.788 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 10 client.54188.objecter _send_command 2 2024-09-10T20:34:56.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 --> v2:172.21.15.176:6800/2372911183 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f69980a4b70 con 0x7f697c043e40 2024-09-10T20:34:56.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f698dffb640 1 -- 172.21.15.153:0/431899275 <== osd.5 v2:172.21.15.176:6800/2372911183 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f69980a4b70 con 0x7f697c043e40 2024-09-10T20:34:56.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f698dffb640 10 client.54188.objecter ms_dispatch 0x7f6998000ba0 command_reply(tid 2: 0 ) 2024-09-10T20:34:56.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f698dffb640 10 client.54188.objecter _finish_command 2 = system:0 2024-09-10T20:34:56.789 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f698dffb640 20 client.54188.objecter put_session s=0x7f697c043790 osd=5 4 2024-09-10T20:34:56.790 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f698dffb640 15 client.54188.objecter _session_command_op_remove 5 2 2024-09-10T20:34:56.790 INFO:teuthology.orchestra.run.smithi153.stdout:{ 2024-09-10T20:34:56.790 INFO:teuthology.orchestra.run.smithi153.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-10T20:34:56.790 INFO:teuthology.orchestra.run.smithi153.stdout:} 2024-09-10T20:34:56.791 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 10 librados: watch_flush enter 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 10 librados: watch_flush exit 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 10 client.54188.objecter close_session for osd.5 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6800/2372911183 conn(0x7f697c043e40 msgr2=0x7f697c0462e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 --2- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6800/2372911183 conn(0x7f697c043e40 0x7f697c0462e0 crc :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 20 client.54188.objecter put_session s=0x7f697c043790 osd=5 1 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 20 client.54188.objecter shutdown clearing up homeless session... 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 10 client.54188.objecter successfully canceled tick 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6812/3003473398 conn(0x7f697c03cef0 msgr2=0x7f697c03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 --2- 172.21.15.153:0/431899275 >> v2:172.21.15.176:6812/3003473398 conn(0x7f697c03cef0 0x7f697c03f3b0 secure :-1 s=READY pgs=3386 cs=0 l=1 rev1=1 crypto rx=0x7f6994009580 tx=0x7f6994009190 comp rx=0 tx=0).stop 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 10 monclient: shutdown 2024-09-10T20:34:56.792 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:56.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6998138db0 msgr2=0x7f699813da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 --2- 172.21.15.153:0/431899275 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f6998138db0 0x7f699813da80 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6990007e70 tx=0x7f699000e4b0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699d8dc640 1 -- 172.21.15.153:0/431899275 reap_dead start 2024-09-10T20:34:56.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 shutdown_connections 2024-09-10T20:34:56.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 >> 172.21.15.153:0/431899275 conn(0x7f69980ab920 msgr2=0x7f699813efa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:56.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 shutdown_connections 2024-09-10T20:34:56.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 -- 172.21.15.153:0/431899275 wait complete. 2024-09-10T20:34:56.793 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.785+0000 7f699f33f640 1 librados: shutdown 2024-09-10T20:34:56.805 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient: get_monmap_and_config 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient: build_initial_monmap 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient: monmap: 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:56.917894+0000 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:56.917894+0000 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:56.918 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:56.919 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:56.919 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:56.919 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:56.919 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 1 Processor -- start 2024-09-10T20:34:56.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 1 -- start start 2024-09-10T20:34:56.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient: init 2024-09-10T20:34:56.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:56.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient: _add_conns ranks=[3,4,2,0,1] 2024-09-10T20:34:56.921 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fab900a41a0 0x7fab900a3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient(hunting): picked mon.noname-a con 0x7fab900a41a0 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fab900a4580 0x7fab900a59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient(hunting): picked mon.noname-c con 0x7fab900a4580 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab900a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient(hunting): picked mon.noname-b con 0x7fab900a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fab900a2570 con 0x7fab900a41a0 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fab90070400 con 0x7fab900a4580 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fab900a6a20 con 0x7fab900a49d0 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:56.922 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab94b93640 10 monclient(hunting): authenticate will time out at 1457.057400s 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fab900a41a0 0x7fab900a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab900a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fab900a41a0 0x7fab900a3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56368/0 (socket says 172.21.15.153:56368) 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 1 -- 172.21.15.153:0/1971103337 learned_addr learned my addr 172.21.15.153:0/1971103337 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 10 monclient(hunting): get_auth_request con 0x7fab900a41a0 auth_method 0 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient(hunting): get_auth_request con 0x7fab900a49d0 auth_method 0 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.923 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.924 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8dd9b640 1 --2- 172.21.15.153:0/1971103337 >> v2:172.21.15.176:3300/0 conn(0x7fab900a4580 0x7fab900a59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8dd9b640 10 monclient(hunting): get_auth_request con 0x7fab900a4580 auth_method 0 2024-09-10T20:34:56.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient(hunting): handle_auth_done global_id 54191 payload 274 2024-09-10T20:34:56.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 1 -- 172.21.15.153:0/1971103337 >> v2:172.21.15.176:3300/0 conn(0x7fab900a4580 msgr2=0x7fab900a59d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:56.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 1 --2- 172.21.15.153:0/1971103337 >> v2:172.21.15.176:3300/0 conn(0x7fab900a4580 0x7fab900a59d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 1 -- 172.21.15.153:0/1971103337 >> v2:172.21.15.153:3300/0 conn(0x7fab900a41a0 msgr2=0x7fab900a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:56.925 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 1 --2- 172.21.15.153:0/1971103337 >> v2:172.21.15.153:3300/0 conn(0x7fab900a41a0 0x7fab900a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient: _finish_hunting 0 2024-09-10T20:34:56.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 1 monclient: found mon.noname-b 2024-09-10T20:34:56.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:56.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8ed9d640 1 -- 172.21.15.153:0/1971103337 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab900a3c80 con 0x7fab900a49d0 2024-09-10T20:34:56.926 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.917+0000 7fab8e59c640 1 --2- 172.21.15.153:0/1971103337 >> v2:172.21.15.153:3300/0 conn(0x7fab900a41a0 0x7fab900a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:56.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8ed9d640 1 -- 172.21.15.153:0/1971103337 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 msgr2=0x7fab900a5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:56.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8ed9d640 1 --2- 172.21.15.153:0/1971103337 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab900a5f10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fab90070720 tx=0x7fab8801c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9848b7483ad17c35 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 1 -- 172.21.15.153:0/1971103337 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab880ad070 con 0x7fab900a49d0 2024-09-10T20:34:56.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:34:56.933 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 10 monclient: dump: 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 10 monclient: _finish_auth 0 2024-09-10T20:34:56.934 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 10 monclient: _check_auth_tickets 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 1 -- 172.21.15.153:0/1971103337 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab880a9670 con 0x7fab900a49d0 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 1 -- 172.21.15.153:0/1971103337 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab880a9a40 con 0x7fab900a49d0 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 5 monclient: authenticate success, global_id 54191 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 4 set_mon_vals no callback set 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab8d59a640 10 monclient: dump: 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.935 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 10 monclient: shutdown 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 -- 172.21.15.153:0/1971103337 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 msgr2=0x7fab900a5f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 --2- 172.21.15.153:0/1971103337 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab900a5f10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fab90070720 tx=0x7fab8801c7f0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 -- 172.21.15.153:0/1971103337 shutdown_connections 2024-09-10T20:34:56.936 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 --2- 172.21.15.153:0/1971103337 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab900a5f10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 --2- 172.21.15.153:0/1971103337 >> v2:172.21.15.176:3300/0 conn(0x7fab900a4580 0x7fab900a59d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 --2- 172.21.15.153:0/1971103337 >> v2:172.21.15.153:3300/0 conn(0x7fab900a41a0 0x7fab900a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 -- 172.21.15.153:0/1971103337 >> 172.21.15.153:0/1971103337 conn(0x7fab900ab7e0 msgr2=0x7fab900a7ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 -- 172.21.15.153:0/1971103337 shutdown_connections 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 -- 172.21.15.153:0/1971103337 wait complete. 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 10 monclient: build_initial_monmap 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 10 monclient: monmap: 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:56.937 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 librados: starting msgr at 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 librados: starting objecter 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 Processor -- start 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 -- start start 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 librados: setting wanted keys 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 1 librados: calling monclient init 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.921+0000 7fab94b93640 10 monclient: init 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab900a41a0 0x7fab901d0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.938 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient(hunting): picked mon.noname-a con 0x7fab900a41a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab900a4580 0x7fab9013e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient(hunting): picked mon.noname-b con 0x7fab900a4580 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab90126400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient(hunting): picked mon.noname-c con 0x7fab900a49d0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fab90142200 con 0x7fab900a41a0 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fab900a6860 con 0x7fab900a4580 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fab90142380 con 0x7fab900a49d0 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:56.939 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient(hunting): authenticate will time out at 1457.064208s 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab90126400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab90126400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:37580/0 (socket says 172.21.15.153:37580) 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 -- 172.21.15.153:0/750831762 learned_addr learned my addr 172.21.15.153:0/750831762 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient(hunting): get_auth_request con 0x7fab900a49d0 auth_method 0 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8e59c640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab900a41a0 0x7fab901d0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8e59c640 10 monclient(hunting): get_auth_request con 0x7fab900a41a0 auth_method 0 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8e59c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8e59c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.940 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8e59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8e59c640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8e59c640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8e59c640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8dd9b640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab900a4580 0x7fab9013e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8dd9b640 10 monclient(hunting): get_auth_request con 0x7fab900a4580 auth_method 0 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8dd9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient(hunting): handle_auth_done global_id 54194 payload 824 2024-09-10T20:34:56.941 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 -- 172.21.15.153:0/750831762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab900a4580 msgr2=0x7fab9013e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab900a4580 0x7fab9013e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 -- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab900a41a0 msgr2=0x7fab901d0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab900a41a0 0x7fab901d0540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient: _finish_hunting 0 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 monclient: found mon.noname-c 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 -- 172.21.15.153:0/750831762 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab90127ec0 con 0x7fab900a49d0 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8e59c640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab900a41a0 0x7fab901d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 -- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 msgr2=0x7fab90126400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8ed9d640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab90126400 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fab8801d090 tx=0x7fab88019590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 client.?.objecter ms_handle_connect 0x7fab900a49d0 2024-09-10T20:34:56.942 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 1 -- 172.21.15.153:0/750831762 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab880c2070 con 0x7fab900a49d0 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 monclient: dump: 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.943 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 monclient: _finish_auth 0 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 monclient: _check_auth_tickets 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 1 -- 172.21.15.153:0/750831762 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab880a9e40 con 0x7fab900a49d0 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 1 -- 172.21.15.153:0/750831762 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab880ad040 con 0x7fab900a49d0 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 5 monclient: authenticate success, global_id 54194 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient: _renew_subs 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab901d0a80 con 0x7fab900a49d0 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab8cd99640 4 set_mon_vals no callback set 2024-09-10T20:34:56.944 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab7effd640 10 monclient: dump: 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:56.945 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 client.54194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient: _renew_subs 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fab90129980 con 0x7fab900a49d0 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.925+0000 7fab94b93640 1 librados: init done 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab94b93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab94b93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab94b93640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab900a3720 con 0x7fab900a49d0 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab7effd640 1 -- 172.21.15.153:0/750831762 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fab880c5070 con 0x7fab900a49d0 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab7effd640 1 --2- 172.21.15.153:0/750831762 >> v2:172.21.15.176:6812/3003473398 conn(0x7fab7003cea0 0x7fab7003f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab7effd640 1 -- 172.21.15.153:0/750831762 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7fab880ea4e0 con 0x7fab900a49d0 2024-09-10T20:34:56.946 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab7effd640 10 client.54194.objecter ms_dispatch 0x7fab90000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:56.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab7effd640 3 client.54194.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:56.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab7effd640 3 client.54194.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:56.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab7effd640 20 client.54194.objecter dump_active .. 0 homeless 2024-09-10T20:34:56.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab8e59c640 1 --2- 172.21.15.153:0/750831762 >> v2:172.21.15.176:6812/3003473398 conn(0x7fab7003cea0 0x7fab7003f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:56.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab8e59c640 10 monclient: get_auth_request con 0x7fab7003cea0 auth_method 0 2024-09-10T20:34:56.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab7effd640 10 client.54194.objecter ms_handle_connect 0x7fab7003cea0 2024-09-10T20:34:56.947 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.929+0000 7fab8e59c640 1 --2- 172.21.15.153:0/750831762 >> v2:172.21.15.176:6812/3003473398 conn(0x7fab7003cea0 0x7fab7003f360 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7fab84009630 tx=0x7fab84009240 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:56.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.933+0000 7fab7effd640 1 -- 172.21.15.153:0/750831762 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fab880b8070 con 0x7fab900a49d0 2024-09-10T20:34:56.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.933+0000 7fab7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:56.948 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:56.933+0000 7fab7effd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:57.100 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.097+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7fab90129ea0 con 0x7fab7003cea0 2024-09-10T20:34:57.101 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.097+0000 7fab7effd640 1 -- 172.21.15.153:0/750831762 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fab90129ea0 con 0x7fab7003cea0 2024-09-10T20:34:57.101 INFO:teuthology.orchestra.run.smithi153.stderr:instructed osd(s) 5 to deep-scrub 2024-09-10T20:34:57.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 10 librados: watch_flush enter 2024-09-10T20:34:57.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 10 librados: watch_flush exit 2024-09-10T20:34:57.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 20 client.54194.objecter shutdown clearing up homeless session... 2024-09-10T20:34:57.103 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 10 client.54194.objecter successfully canceled tick 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 >> v2:172.21.15.176:6812/3003473398 conn(0x7fab7003cea0 msgr2=0x7fab7003f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 --2- 172.21.15.153:0/750831762 >> v2:172.21.15.176:6812/3003473398 conn(0x7fab7003cea0 0x7fab7003f360 secure :-1 s=READY pgs=3387 cs=0 l=1 rev1=1 crypto rx=0x7fab84009630 tx=0x7fab84009240 comp rx=0 tx=0).stop 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 10 monclient: shutdown 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 msgr2=0x7fab90126400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab90126400 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fab8801d090 tx=0x7fab88019590 comp rx=0 tx=0).stop 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 shutdown_connections 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 --2- 172.21.15.153:0/750831762 >> v2:172.21.15.176:6812/3003473398 conn(0x7fab7003cea0 0x7fab7003f360 unknown :-1 s=CLOSED pgs=3387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fab900a49d0 0x7fab90126400 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fab900a4580 0x7fab9013e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 --2- 172.21.15.153:0/750831762 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fab900a41a0 0x7fab901d0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 >> 172.21.15.153:0/750831762 conn(0x7fab900ab7e0 msgr2=0x7fab900a58d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:57.104 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 shutdown_connections 2024-09-10T20:34:57.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 -- 172.21.15.153:0/750831762 wait complete. 2024-09-10T20:34:57.105 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.101+0000 7fab94b93640 1 librados: shutdown 2024-09-10T20:34:57.119 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.229+0000 7f45d7c20640 10 monclient: get_monmap_and_config 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.229+0000 7f45d7c20640 10 monclient: build_initial_monmap 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.229+0000 7f45d7c20640 10 monclient: monmap: 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:34:57.232810+0000 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:34:57.232810+0000 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:34:57.233 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:34:57.234 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:34:57.234 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:34:57.234 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:57.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.229+0000 7f45d7c20640 1 Processor -- start 2024-09-10T20:34:57.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.229+0000 7f45d7c20640 1 -- start start 2024-09-10T20:34:57.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.229+0000 7f45d7c20640 10 monclient: init 2024-09-10T20:34:57.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:57.236 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient: _add_conns ranks=[3,4,2,1,0] 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45d00a2e70 0x7f45d00a3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient(hunting): picked mon.noname-a con 0x7f45d00a2e70 addr v2:172.21.15.153:3300/0 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7f45d00a3790 0x7f45d01385e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient(hunting): picked mon.noname-c con 0x7f45d00a3790 addr v2:172.21.15.176:3300/0 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0138fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient(hunting): picked mon.noname-b con 0x7f45d0138b20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7f45d009b740 con 0x7f45d00a2e70 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7f45d009e290 con 0x7f45d00a3790 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45d009b100 con 0x7f45d0138b20 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:57.237 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d7c20640 10 monclient(hunting): authenticate will time out at 1457.372564s 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45d00a2e70 0x7f45d00a3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d61bd640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0138fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7f45d00a2e70 0x7f45d00a3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56384/0 (socket says 172.21.15.153:56384) 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 1 -- 172.21.15.153:0/1986644141 learned_addr learned my addr 172.21.15.153:0/1986644141 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d61bd640 10 monclient(hunting): get_auth_request con 0x7f45d0138b20 auth_method 0 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d61bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d61bd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d61bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient(hunting): get_auth_request con 0x7f45d00a2e70 auth_method 0 2024-09-10T20:34:57.238 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d61bd640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d61bd640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d61bd640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d51bb640 1 --2- 172.21.15.153:0/1986644141 >> v2:172.21.15.176:3300/0 conn(0x7f45d00a3790 0x7f45d01385e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient(hunting): handle_auth_done global_id 34458 payload 274 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 1 -- 172.21.15.153:0/1986644141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 msgr2=0x7f45d0138fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 1 --2- 172.21.15.153:0/1986644141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0138fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 1 -- 172.21.15.153:0/1986644141 >> v2:172.21.15.176:3300/0 conn(0x7f45d00a3790 msgr2=0x7f45d01385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:34:57.239 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 1 --2- 172.21.15.153:0/1986644141 >> v2:172.21.15.176:3300/0 conn(0x7f45d00a3790 0x7f45d01385e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient: _finish_hunting 0 2024-09-10T20:34:57.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 1 monclient: found mon.noname-a 2024-09-10T20:34:57.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:57.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:57.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d59bc640 1 -- 172.21.15.153:0/1986644141 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d009b460 con 0x7f45d00a2e70 2024-09-10T20:34:57.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d51bb640 10 monclient: get_auth_request con 0x7f45d00a3790 auth_method 0 2024-09-10T20:34:57.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d51bb640 1 --2- 172.21.15.153:0/1986644141 >> v2:172.21.15.176:3300/0 conn(0x7f45d00a3790 0x7f45d01385e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:34:57.240 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.233+0000 7f45d61bd640 1 --2- 172.21.15.153:0/1986644141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:34:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 -- 172.21.15.153:0/1986644141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 msgr2=0x7f45d00a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 --2- 172.21.15.153:0/1986644141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 0x7f45d00a3250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f45d006fbc0 tx=0x7f45c801c3e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4b01ab6989686e8 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 1 -- 172.21.15.153:0/1986644141 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80b0070 con 0x7f45d00a2e70 2024-09-10T20:34:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:57.243 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 10 monclient: dump: 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 10 monclient: _finish_auth 0 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 10 monclient: _check_auth_tickets 2024-09-10T20:34:57.244 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 1 -- 172.21.15.153:0/1986644141 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c80ab600 con 0x7f45d00a2e70 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 1 -- 172.21.15.153:0/1986644141 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80ab9d0 con 0x7f45d00a2e70 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 5 monclient: authenticate success, global_id 34458 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient: get_monmap_and_config success 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 4 set_mon_vals no callback set 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d49ba640 10 monclient: dump: 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:57.245 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient: shutdown 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 -- 172.21.15.153:0/1986644141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 msgr2=0x7f45d00a3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:57.246 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 --2- 172.21.15.153:0/1986644141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 0x7f45d00a3250 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f45d006fbc0 tx=0x7f45c801c3e0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 -- 172.21.15.153:0/1986644141 shutdown_connections 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 --2- 172.21.15.153:0/1986644141 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0138fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 --2- 172.21.15.153:0/1986644141 >> v2:172.21.15.176:3300/0 conn(0x7f45d00a3790 0x7f45d01385e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 --2- 172.21.15.153:0/1986644141 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 0x7f45d00a3250 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 -- 172.21.15.153:0/1986644141 >> 172.21.15.153:0/1986644141 conn(0x7f45d00a3f60 msgr2=0x7f45d00a4340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 -- 172.21.15.153:0/1986644141 shutdown_connections 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 -- 172.21.15.153:0/1986644141 wait complete. 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient: build_initial_monmap 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient: monmap: 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:34:57.247 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 librados: starting msgr at 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 librados: starting objecter 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 Processor -- start 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 -- start start 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 librados: setting wanted keys 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 librados: calling monclient init 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient: init 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient: _reopen_session rank -1 2024-09-10T20:34:57.248 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 0x7f45d01d0470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient(hunting): picked mon.noname-a con 0x7f45d00a2e70 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45d00a3790 0x7f45d00982c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient(hunting): picked mon.noname-b con 0x7f45d00a3790 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0098800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient(hunting): picked mon.noname-c con 0x7f45d0138b20 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7f45d006d9b0 con 0x7f45d00a2e70 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7f45d006d830 con 0x7f45d00a3790 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient(hunting): start opening mon connection 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7f45d00a4c50 con 0x7f45d0138b20 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient(hunting): _renew_subs 2024-09-10T20:34:57.249 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d7c20640 10 monclient(hunting): authenticate will time out at 1457.379553s 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 0x7f45d01d0470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 0x7f45d01d0470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:56400/0 (socket says 172.21.15.153:56400) 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 -- 172.21.15.153:0/4252799432 learned_addr learned my addr 172.21.15.153:0/4252799432 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient(hunting): get_auth_request con 0x7f45d00a2e70 auth_method 0 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d61bd640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0098800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d61bd640 10 monclient(hunting): get_auth_request con 0x7f45d0138b20 auth_method 0 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d61bd640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:34:57.250 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d61bd640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d61bd640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient(hunting): handle_auth_done global_id 34461 payload 824 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 -- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 msgr2=0x7f45d0098800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0098800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 -- 172.21.15.153:0/4252799432 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45d00a3790 msgr2=0x7f45d00982c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d51bb640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45d00a3790 0x7f45d00982c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45d00a3790 0x7f45d00982c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient: _finish_hunting 0 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 monclient: found mon.noname-a 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d59bc640 1 -- 172.21.15.153:0/4252799432 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45d008fd20 con 0x7f45d00a2e70 2024-09-10T20:34:57.251 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.237+0000 7f45d61bd640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0098800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d59bc640 1 -- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 msgr2=0x7f45d01d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d59bc640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 0x7f45d01d0470 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f45c801aa90 tx=0x7f45c8019180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 client.?.objecter ms_handle_connect 0x7f45d00a2e70 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 client.?.objecter resend_mon_ops 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 1 -- 172.21.15.153:0/4252799432 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80b0040 con 0x7f45d00a2e70 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 monclient: dump: 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:57.258 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 monclient: _finish_auth 0 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 monclient: _check_auth_tickets 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 1 -- 172.21.15.153:0/4252799432 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45c80aa5b0 con 0x7f45d00a2e70 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 monclient: handle_config config(0 keys) 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 1 -- 172.21.15.153:0/4252799432 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45c80aa9a0 con 0x7f45d00a2e70 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:34:57.259 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 10 monclient: dump: 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c77fe640 4 set_mon_vals no callback set 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 5 monclient: authenticate success, global_id 34461 2024-09-10T20:34:57.260 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 10 monclient: _renew_subs 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45d01d09b0 con 0x7f45d00a2e70 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 10 client.34461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 10 monclient: _renew_subs 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45d0098d40 con 0x7f45d00a2e70 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 1 librados: init done 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:57.261 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 10 monclient: _send_mon_message to mon.a at v2:172.21.15.153:3300/0 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45d01385e0 con 0x7f45d00a2e70 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.241+0000 7f45c67fc640 1 -- 172.21.15.153:0/4252799432 <== mon.0 v2:172.21.15.153:3300/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7f45c80aab40 con 0x7f45d00a2e70 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45c67fc640 1 --2- 172.21.15.153:0/4252799432 >> v2:172.21.15.176:6812/3003473398 conn(0x7f45bc03cea0 0x7f45bc03f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45c67fc640 1 -- 172.21.15.153:0/4252799432 <== mon.0 v2:172.21.15.153:3300/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7f45c80e9330 con 0x7f45d00a2e70 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45c67fc640 10 client.34461.objecter ms_dispatch 0x7f45d0000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45c67fc640 3 client.34461.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45c67fc640 3 client.34461.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45c67fc640 20 client.34461.objecter dump_active .. 0 homeless 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45d51bb640 1 --2- 172.21.15.153:0/4252799432 >> v2:172.21.15.176:6812/3003473398 conn(0x7f45bc03cea0 0x7f45bc03f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45d51bb640 10 monclient: get_auth_request con 0x7f45bc03cea0 auth_method 0 2024-09-10T20:34:57.262 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45c67fc640 10 client.34461.objecter ms_handle_connect 0x7f45bc03cea0 2024-09-10T20:34:57.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.245+0000 7f45d51bb640 1 --2- 172.21.15.153:0/4252799432 >> v2:172.21.15.176:6812/3003473398 conn(0x7f45bc03cea0 0x7f45bc03f360 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f45d013ea10 tx=0x7f45cc0084d0 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:34:57.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.249+0000 7f45c67fc640 1 -- 172.21.15.153:0/4252799432 <== mon.0 v2:172.21.15.153:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7f45c80160f0 con 0x7f45d00a2e70 2024-09-10T20:34:57.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.249+0000 7f45c67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:34:57.263 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.249+0000 7f45c67fc640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:34:57.406 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.401+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f45d013e480 con 0x7f45bc03cea0 2024-09-10T20:34:57.411 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.409+0000 7f45c67fc640 1 -- 172.21.15.153:0/4252799432 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114145 (secure 0 0 0) 0x7f45d013e480 con 0x7f45bc03cea0 2024-09-10T20:34:57.413 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:34:57.414 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 10 librados: watch_flush enter 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 10 librados: watch_flush exit 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 20 client.34461.objecter shutdown clearing up homeless session... 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 10 client.34461.objecter successfully canceled tick 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 >> v2:172.21.15.176:6812/3003473398 conn(0x7f45bc03cea0 msgr2=0x7f45bc03f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 --2- 172.21.15.153:0/4252799432 >> v2:172.21.15.176:6812/3003473398 conn(0x7f45bc03cea0 0x7f45bc03f360 secure :-1 s=READY pgs=3388 cs=0 l=1 rev1=1 crypto rx=0x7f45d013ea10 tx=0x7f45cc0084d0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 10 monclient: shutdown 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 msgr2=0x7f45d01d0470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 0x7f45d01d0470 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f45c801aa90 tx=0x7f45c8019180 comp rx=0 tx=0).stop 2024-09-10T20:34:57.416 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 shutdown_connections 2024-09-10T20:34:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 --2- 172.21.15.153:0/4252799432 >> v2:172.21.15.176:6812/3003473398 conn(0x7f45bc03cea0 0x7f45bc03f360 unknown :-1 s=CLOSED pgs=3388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7f45d0138b20 0x7f45d0098800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7f45d00a3790 0x7f45d00982c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 --2- 172.21.15.153:0/4252799432 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7f45d00a2e70 0x7f45d01d0470 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:34:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 >> 172.21.15.153:0/4252799432 conn(0x7f45d00a3f60 msgr2=0x7f45d00a5360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:34:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 shutdown_connections 2024-09-10T20:34:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 -- 172.21.15.153:0/4252799432 wait complete. 2024-09-10T20:34:57.417 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:34:57.413+0000 7f45d7c20640 1 librados: shutdown 2024-09-10T20:34:57.430 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":574,"stamp":"2024-09-10T20:34:57.251453+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":449376,"kb_used_data":28984,"kb_used_omap":9,"kb_used_meta":420214,"kb_avail":565781664,"statfs":{"total":579820584960,"available":579360423936,"internally_reserved":0,"allocated":29679616,"data_stored":15420662,"data_compressed":596482,"data_compressed_allocated":966656,"data_compressed_original":3112960,"omap_allocated":9567,"internal_metadata":430299809},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003843"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.828961+0000","last_change":"2024-09-10T20:27:17.198257+0000","last_active":"2024-09-10T20:33:29.828961+0000","last_peered":"2024-09-10T20:33:29.828961+0000","last_clean":"2024-09-10T20:33:29.828961+0000","last_became_active":"2024-09-10T20:27:17.197839+0000","last_became_peered":"2024-09-10T20:27:17.197839+0000","last_unstale":"2024-09-10T20:33:29.828961+0000","last_undegraded":"2024-09-10T20:33:29.828961+0000","last_fullsized":"2024-09-10T20:33:29.828961+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:49:15.959864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500272+0000","last_change":"2024-09-10T20:27:17.223318+0000","last_active":"2024-09-10T20:33:29.500272+0000","last_peered":"2024-09-10T20:33:29.500272+0000","last_clean":"2024-09-10T20:33:29.500272+0000","last_became_active":"2024-09-10T20:27:17.219498+0000","last_became_peered":"2024-09-10T20:27:17.219498+0000","last_unstale":"2024-09-10T20:33:29.500272+0000","last_undegraded":"2024-09-10T20:33:29.500272+0000","last_fullsized":"2024-09-10T20:33:29.500272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:20:12.081089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498568+0000","last_change":"2024-09-10T20:27:17.202919+0000","last_active":"2024-09-10T20:33:29.498568+0000","last_peered":"2024-09-10T20:33:29.498568+0000","last_clean":"2024-09-10T20:33:29.498568+0000","last_became_active":"2024-09-10T20:27:17.202373+0000","last_became_peered":"2024-09-10T20:27:17.202373+0000","last_unstale":"2024-09-10T20:33:29.498568+0000","last_undegraded":"2024-09-10T20:33:29.498568+0000","last_fullsized":"2024-09-10T20:33:29.498568+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:42:27.202536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500829+0000","last_change":"2024-09-10T20:27:17.218042+0000","last_active":"2024-09-10T20:33:29.500829+0000","last_peered":"2024-09-10T20:33:29.500829+0000","last_clean":"2024-09-10T20:33:29.500829+0000","last_became_active":"2024-09-10T20:27:17.217517+0000","last_became_peered":"2024-09-10T20:27:17.217517+0000","last_unstale":"2024-09-10T20:33:29.500829+0000","last_undegraded":"2024-09-10T20:33:29.500829+0000","last_fullsized":"2024-09-10T20:33:29.500829+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:32:53.695011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829236+0000","last_change":"2024-09-10T20:27:17.282375+0000","last_active":"2024-09-10T20:33:29.829236+0000","last_peered":"2024-09-10T20:33:29.829236+0000","last_clean":"2024-09-10T20:33:29.829236+0000","last_became_active":"2024-09-10T20:27:17.281836+0000","last_became_peered":"2024-09-10T20:27:17.281836+0000","last_unstale":"2024-09-10T20:33:29.829236+0000","last_undegraded":"2024-09-10T20:33:29.829236+0000","last_fullsized":"2024-09-10T20:33:29.829236+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:51:27.168991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498729+0000","last_change":"2024-09-10T20:27:17.213190+0000","last_active":"2024-09-10T20:33:29.498729+0000","last_peered":"2024-09-10T20:33:29.498729+0000","last_clean":"2024-09-10T20:33:29.498729+0000","last_became_active":"2024-09-10T20:27:17.212620+0000","last_became_peered":"2024-09-10T20:27:17.212620+0000","last_unstale":"2024-09-10T20:33:29.498729+0000","last_undegraded":"2024-09-10T20:33:29.498729+0000","last_fullsized":"2024-09-10T20:33:29.498729+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:58:48.977530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498751+0000","last_change":"2024-09-10T20:27:17.288663+0000","last_active":"2024-09-10T20:33:29.498751+0000","last_peered":"2024-09-10T20:33:29.498751+0000","last_clean":"2024-09-10T20:33:29.498751+0000","last_became_active":"2024-09-10T20:27:17.287973+0000","last_became_peered":"2024-09-10T20:27:17.287973+0000","last_unstale":"2024-09-10T20:33:29.498751+0000","last_undegraded":"2024-09-10T20:33:29.498751+0000","last_fullsized":"2024-09-10T20:33:29.498751+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:16:25.413353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874217+0000","last_change":"2024-09-10T20:27:17.210079+0000","last_active":"2024-09-10T20:33:29.874217+0000","last_peered":"2024-09-10T20:33:29.874217+0000","last_clean":"2024-09-10T20:33:29.874217+0000","last_became_active":"2024-09-10T20:27:17.209693+0000","last_became_peered":"2024-09-10T20:27:17.209693+0000","last_unstale":"2024-09-10T20:33:29.874217+0000","last_undegraded":"2024-09-10T20:33:29.874217+0000","last_fullsized":"2024-09-10T20:33:29.874217+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:56:44.983674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500800+0000","last_change":"2024-09-10T20:27:17.249972+0000","last_active":"2024-09-10T20:33:29.500800+0000","last_peered":"2024-09-10T20:33:29.500800+0000","last_clean":"2024-09-10T20:33:29.500800+0000","last_became_active":"2024-09-10T20:27:17.249683+0000","last_became_peered":"2024-09-10T20:27:17.249683+0000","last_unstale":"2024-09-10T20:33:29.500800+0000","last_undegraded":"2024-09-10T20:33:29.500800+0000","last_fullsized":"2024-09-10T20:33:29.500800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:02:11.684811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498432+0000","last_change":"2024-09-10T20:27:17.248264+0000","last_active":"2024-09-10T20:33:29.498432+0000","last_peered":"2024-09-10T20:33:29.498432+0000","last_clean":"2024-09-10T20:33:29.498432+0000","last_became_active":"2024-09-10T20:27:17.247865+0000","last_became_peered":"2024-09-10T20:27:17.247865+0000","last_unstale":"2024-09-10T20:33:29.498432+0000","last_undegraded":"2024-09-10T20:33:29.498432+0000","last_fullsized":"2024-09-10T20:33:29.498432+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:06:33.643650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829007+0000","last_change":"2024-09-10T20:27:17.203381+0000","last_active":"2024-09-10T20:33:29.829007+0000","last_peered":"2024-09-10T20:33:29.829007+0000","last_clean":"2024-09-10T20:33:29.829007+0000","last_became_active":"2024-09-10T20:27:17.203002+0000","last_became_peered":"2024-09-10T20:27:17.203002+0000","last_unstale":"2024-09-10T20:33:29.829007+0000","last_undegraded":"2024-09-10T20:33:29.829007+0000","last_fullsized":"2024-09-10T20:33:29.829007+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:32:20.452158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873964+0000","last_change":"2024-09-10T20:27:17.298627+0000","last_active":"2024-09-10T20:33:29.873964+0000","last_peered":"2024-09-10T20:33:29.873964+0000","last_clean":"2024-09-10T20:33:29.873964+0000","last_became_active":"2024-09-10T20:27:17.298309+0000","last_became_peered":"2024-09-10T20:27:17.298309+0000","last_unstale":"2024-09-10T20:33:29.873964+0000","last_undegraded":"2024-09-10T20:33:29.873964+0000","last_fullsized":"2024-09-10T20:33:29.873964+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:50:55.361013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":596,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500685+0000","last_change":"2024-09-10T20:26:21.773251+0000","last_active":"2024-09-10T20:33:29.500685+0000","last_peered":"2024-09-10T20:33:29.500685+0000","last_clean":"2024-09-10T20:33:29.500685+0000","last_became_active":"2024-09-10T20:26:19.356844+0000","last_became_peered":"2024-09-10T20:26:19.356844+0000","last_unstale":"2024-09-10T20:33:29.500685+0000","last_undegraded":"2024-09-10T20:33:29.500685+0000","last_fullsized":"2024-09-10T20:33:29.500685+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:23:23.413224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873697+0000","last_change":"2024-09-10T20:27:17.206032+0000","last_active":"2024-09-10T20:33:29.873697+0000","last_peered":"2024-09-10T20:33:29.873697+0000","last_clean":"2024-09-10T20:33:29.873697+0000","last_became_active":"2024-09-10T20:27:17.205464+0000","last_became_peered":"2024-09-10T20:27:17.205464+0000","last_unstale":"2024-09-10T20:33:29.873697+0000","last_undegraded":"2024-09-10T20:33:29.873697+0000","last_fullsized":"2024-09-10T20:33:29.873697+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:16:33.219827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":593,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829238+0000","last_change":"2024-09-10T20:26:21.381255+0000","last_active":"2024-09-10T20:33:29.829238+0000","last_peered":"2024-09-10T20:33:29.829238+0000","last_clean":"2024-09-10T20:33:29.829238+0000","last_became_active":"2024-09-10T20:26:19.363220+0000","last_became_peered":"2024-09-10T20:26:19.363220+0000","last_unstale":"2024-09-10T20:33:29.829238+0000","last_undegraded":"2024-09-10T20:33:29.829238+0000","last_fullsized":"2024-09-10T20:33:29.829238+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:24:09.279291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096679600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"23'7","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500753+0000","last_change":"2024-09-10T20:27:17.217628+0000","last_active":"2024-09-10T20:33:29.500753+0000","last_peered":"2024-09-10T20:33:29.500753+0000","last_clean":"2024-09-10T20:33:29.500753+0000","last_became_active":"2024-09-10T20:27:17.211037+0000","last_became_peered":"2024-09-10T20:27:17.211037+0000","last_unstale":"2024-09-10T20:33:29.500753+0000","last_undegraded":"2024-09-10T20:33:29.500753+0000","last_fullsized":"2024-09-10T20:33:29.500753+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:06:47.151425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":597,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500610+0000","last_change":"2024-09-10T20:26:21.773020+0000","last_active":"2024-09-10T20:33:29.500610+0000","last_peered":"2024-09-10T20:33:29.500610+0000","last_clean":"2024-09-10T20:33:29.500610+0000","last_became_active":"2024-09-10T20:26:19.363294+0000","last_became_peered":"2024-09-10T20:26:19.363294+0000","last_unstale":"2024-09-10T20:33:29.500610+0000","last_undegraded":"2024-09-10T20:33:29.500610+0000","last_fullsized":"2024-09-10T20:33:29.500610+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:39:54.300619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053952999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.875890+0000","last_change":"2024-09-10T20:27:17.298070+0000","last_active":"2024-09-10T20:33:29.875890+0000","last_peered":"2024-09-10T20:33:29.875890+0000","last_clean":"2024-09-10T20:33:29.875890+0000","last_became_active":"2024-09-10T20:27:17.297745+0000","last_became_peered":"2024-09-10T20:27:17.297745+0000","last_unstale":"2024-09-10T20:33:29.875890+0000","last_undegraded":"2024-09-10T20:33:29.875890+0000","last_fullsized":"2024-09-10T20:33:29.875890+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:30:22.598793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874003+0000","last_change":"2024-09-10T20:26:21.378377+0000","last_active":"2024-09-10T20:33:29.874003+0000","last_peered":"2024-09-10T20:33:29.874003+0000","last_clean":"2024-09-10T20:33:29.874003+0000","last_became_active":"2024-09-10T20:26:19.359961+0000","last_became_peered":"2024-09-10T20:26:19.359961+0000","last_unstale":"2024-09-10T20:33:29.874003+0000","last_undegraded":"2024-09-10T20:33:29.874003+0000","last_fullsized":"2024-09-10T20:33:29.874003+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:21:01.940376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829065+0000","last_change":"2024-09-10T20:27:17.282394+0000","last_active":"2024-09-10T20:33:29.829065+0000","last_peered":"2024-09-10T20:33:29.829065+0000","last_clean":"2024-09-10T20:33:29.829065+0000","last_became_active":"2024-09-10T20:27:17.281815+0000","last_became_peered":"2024-09-10T20:27:17.281815+0000","last_unstale":"2024-09-10T20:33:29.829065+0000","last_undegraded":"2024-09-10T20:33:29.829065+0000","last_fullsized":"2024-09-10T20:33:29.829065+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:36:49.830688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":597,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876148+0000","last_change":"2024-09-10T20:26:21.386697+0000","last_active":"2024-09-10T20:33:29.876148+0000","last_peered":"2024-09-10T20:33:29.876148+0000","last_clean":"2024-09-10T20:33:29.876148+0000","last_became_active":"2024-09-10T20:26:19.360427+0000","last_became_peered":"2024-09-10T20:26:19.360427+0000","last_unstale":"2024-09-10T20:33:29.876148+0000","last_undegraded":"2024-09-10T20:33:29.876148+0000","last_fullsized":"2024-09-10T20:33:29.876148+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:47:53.136006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020581079999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873800+0000","last_change":"2024-09-10T20:27:17.282845+0000","last_active":"2024-09-10T20:33:29.873800+0000","last_peered":"2024-09-10T20:33:29.873800+0000","last_clean":"2024-09-10T20:33:29.873800+0000","last_became_active":"2024-09-10T20:27:17.282600+0000","last_became_peered":"2024-09-10T20:27:17.282600+0000","last_unstale":"2024-09-10T20:33:29.873800+0000","last_undegraded":"2024-09-10T20:33:29.873800+0000","last_fullsized":"2024-09-10T20:33:29.873800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:10:38.719966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":603,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873765+0000","last_change":"2024-09-10T20:26:21.575187+0000","last_active":"2024-09-10T20:33:29.873765+0000","last_peered":"2024-09-10T20:33:29.873765+0000","last_clean":"2024-09-10T20:33:29.873765+0000","last_became_active":"2024-09-10T20:26:19.362858+0000","last_became_peered":"2024-09-10T20:26:19.362858+0000","last_unstale":"2024-09-10T20:33:29.873765+0000","last_undegraded":"2024-09-10T20:33:29.873765+0000","last_fullsized":"2024-09-10T20:33:29.873765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:22:17.261036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00078714099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829346+0000","last_change":"2024-09-10T20:27:17.214784+0000","last_active":"2024-09-10T20:33:29.829346+0000","last_peered":"2024-09-10T20:33:29.829346+0000","last_clean":"2024-09-10T20:33:29.829346+0000","last_became_active":"2024-09-10T20:27:17.214428+0000","last_became_peered":"2024-09-10T20:27:17.214428+0000","last_unstale":"2024-09-10T20:33:29.829346+0000","last_undegraded":"2024-09-10T20:33:29.829346+0000","last_fullsized":"2024-09-10T20:33:29.829346+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T20:50:15.983575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":610,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498452+0000","last_change":"2024-09-10T20:26:21.378915+0000","last_active":"2024-09-10T20:33:29.498452+0000","last_peered":"2024-09-10T20:33:29.498452+0000","last_clean":"2024-09-10T20:33:29.498452+0000","last_became_active":"2024-09-10T20:26:19.362158+0000","last_became_peered":"2024-09-10T20:26:19.362158+0000","last_unstale":"2024-09-10T20:33:29.498452+0000","last_undegraded":"2024-09-10T20:33:29.498452+0000","last_fullsized":"2024-09-10T20:33:29.498452+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:11:50.708554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010122729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876226+0000","last_change":"2024-09-10T20:27:17.244860+0000","last_active":"2024-09-10T20:33:29.876226+0000","last_peered":"2024-09-10T20:33:29.876226+0000","last_clean":"2024-09-10T20:33:29.876226+0000","last_became_active":"2024-09-10T20:27:17.243935+0000","last_became_peered":"2024-09-10T20:27:17.243935+0000","last_unstale":"2024-09-10T20:33:29.876226+0000","last_undegraded":"2024-09-10T20:33:29.876226+0000","last_fullsized":"2024-09-10T20:33:29.876226+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:01:11.085907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":598,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500566+0000","last_change":"2024-09-10T20:26:21.773025+0000","last_active":"2024-09-10T20:33:29.500566+0000","last_peered":"2024-09-10T20:33:29.500566+0000","last_clean":"2024-09-10T20:33:29.500566+0000","last_became_active":"2024-09-10T20:26:19.359775+0000","last_became_peered":"2024-09-10T20:26:19.359775+0000","last_unstale":"2024-09-10T20:33:29.500566+0000","last_undegraded":"2024-09-10T20:33:29.500566+0000","last_fullsized":"2024-09-10T20:33:29.500566+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:18.337408+0000","last_clean_scrub_stamp":"2024-09-10T20:26:18.337408+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:43:35.427907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054594699999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"16'72","reported_seq":688,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500496+0000","last_change":"2024-09-10T20:26:14.579551+0000","last_active":"2024-09-10T20:33:29.500496+0000","last_peered":"2024-09-10T20:33:29.500496+0000","last_clean":"2024-09-10T20:33:29.500496+0000","last_became_active":"2024-09-10T20:26:14.577602+0000","last_became_peered":"2024-09-10T20:26:14.577602+0000","last_unstale":"2024-09-10T20:33:29.500496+0000","last_undegraded":"2024-09-10T20:33:29.500496+0000","last_fullsized":"2024-09-10T20:33:29.500496+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:26:13.287430+0000","last_clean_scrub_stamp":"2024-09-10T20:26:13.287430+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:52:41.463707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.875916+0000","last_change":"2024-09-10T20:27:17.297570+0000","last_active":"2024-09-10T20:33:29.875916+0000","last_peered":"2024-09-10T20:33:29.875916+0000","last_clean":"2024-09-10T20:33:29.875916+0000","last_became_active":"2024-09-10T20:27:17.295709+0000","last_became_peered":"2024-09-10T20:27:17.295709+0000","last_unstale":"2024-09-10T20:33:29.875916+0000","last_undegraded":"2024-09-10T20:33:29.875916+0000","last_fullsized":"2024-09-10T20:33:29.875916+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:37.519789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":594,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.498888+0000","last_change":"2024-09-10T20:27:17.219710+0000","last_active":"2024-09-10T20:33:29.498888+0000","last_peered":"2024-09-10T20:33:29.498888+0000","last_clean":"2024-09-10T20:33:29.498888+0000","last_became_active":"2024-09-10T20:27:17.217973+0000","last_became_peered":"2024-09-10T20:27:17.217973+0000","last_unstale":"2024-09-10T20:33:29.498888+0000","last_undegraded":"2024-09-10T20:33:29.498888+0000","last_fullsized":"2024-09-10T20:33:29.498888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:17:55.139486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500679+0000","last_change":"2024-09-10T20:27:17.318458+0000","last_active":"2024-09-10T20:33:29.500679+0000","last_peered":"2024-09-10T20:33:29.500679+0000","last_clean":"2024-09-10T20:33:29.500679+0000","last_became_active":"2024-09-10T20:27:17.318202+0000","last_became_peered":"2024-09-10T20:27:17.318202+0000","last_unstale":"2024-09-10T20:33:29.500679+0000","last_undegraded":"2024-09-10T20:33:29.500679+0000","last_fullsized":"2024-09-10T20:33:29.500679+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:35:49.111756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873865+0000","last_change":"2024-09-10T20:27:17.240732+0000","last_active":"2024-09-10T20:33:29.873865+0000","last_peered":"2024-09-10T20:33:29.873865+0000","last_clean":"2024-09-10T20:33:29.873865+0000","last_became_active":"2024-09-10T20:27:17.240435+0000","last_became_peered":"2024-09-10T20:27:17.240435+0000","last_unstale":"2024-09-10T20:33:29.873865+0000","last_undegraded":"2024-09-10T20:33:29.873865+0000","last_fullsized":"2024-09-10T20:33:29.873865+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:54:42.900319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874276+0000","last_change":"2024-09-10T20:27:17.230157+0000","last_active":"2024-09-10T20:33:29.874276+0000","last_peered":"2024-09-10T20:33:29.874276+0000","last_clean":"2024-09-10T20:33:29.874276+0000","last_became_active":"2024-09-10T20:27:17.229886+0000","last_became_peered":"2024-09-10T20:27:17.229886+0000","last_unstale":"2024-09-10T20:33:29.874276+0000","last_undegraded":"2024-09-10T20:33:29.874276+0000","last_fullsized":"2024-09-10T20:33:29.874276+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:08:40.265668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874302+0000","last_change":"2024-09-10T20:27:17.214355+0000","last_active":"2024-09-10T20:33:29.874302+0000","last_peered":"2024-09-10T20:33:29.874302+0000","last_clean":"2024-09-10T20:33:29.874302+0000","last_became_active":"2024-09-10T20:27:17.213875+0000","last_became_peered":"2024-09-10T20:27:17.213875+0000","last_unstale":"2024-09-10T20:33:29.874302+0000","last_undegraded":"2024-09-10T20:33:29.874302+0000","last_fullsized":"2024-09-10T20:33:29.874302+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:04:55.411553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873798+0000","last_change":"2024-09-10T20:27:17.288406+0000","last_active":"2024-09-10T20:33:29.873798+0000","last_peered":"2024-09-10T20:33:29.873798+0000","last_clean":"2024-09-10T20:33:29.873798+0000","last_became_active":"2024-09-10T20:27:17.288092+0000","last_became_peered":"2024-09-10T20:27:17.288092+0000","last_unstale":"2024-09-10T20:33:29.873798+0000","last_undegraded":"2024-09-10T20:33:29.873798+0000","last_fullsized":"2024-09-10T20:33:29.873798+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:28:47.584995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.873374+0000","last_change":"2024-09-10T20:27:17.210249+0000","last_active":"2024-09-10T20:33:29.873374+0000","last_peered":"2024-09-10T20:33:29.873374+0000","last_clean":"2024-09-10T20:33:29.873374+0000","last_became_active":"2024-09-10T20:27:17.209363+0000","last_became_peered":"2024-09-10T20:27:17.209363+0000","last_unstale":"2024-09-10T20:33:29.873374+0000","last_undegraded":"2024-09-10T20:33:29.873374+0000","last_fullsized":"2024-09-10T20:33:29.873374+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:13:19.942713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500468+0000","last_change":"2024-09-10T20:27:17.225434+0000","last_active":"2024-09-10T20:33:29.500468+0000","last_peered":"2024-09-10T20:33:29.500468+0000","last_clean":"2024-09-10T20:33:29.500468+0000","last_became_active":"2024-09-10T20:27:17.224230+0000","last_became_peered":"2024-09-10T20:27:17.224230+0000","last_unstale":"2024-09-10T20:33:29.500468+0000","last_undegraded":"2024-09-10T20:33:29.500468+0000","last_fullsized":"2024-09-10T20:33:29.500468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:49:09.800954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":581,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.500670+0000","last_change":"2024-09-10T20:27:17.217944+0000","last_active":"2024-09-10T20:33:29.500670+0000","last_peered":"2024-09-10T20:33:29.500670+0000","last_clean":"2024-09-10T20:33:29.500670+0000","last_became_active":"2024-09-10T20:27:17.217107+0000","last_became_peered":"2024-09-10T20:27:17.217107+0000","last_unstale":"2024-09-10T20:33:29.500670+0000","last_undegraded":"2024-09-10T20:33:29.500670+0000","last_fullsized":"2024-09-10T20:33:29.500670+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:08:52.708045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":578,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.829272+0000","last_change":"2024-09-10T20:27:17.283023+0000","last_active":"2024-09-10T20:33:29.829272+0000","last_peered":"2024-09-10T20:33:29.829272+0000","last_clean":"2024-09-10T20:33:29.829272+0000","last_became_active":"2024-09-10T20:27:17.282074+0000","last_became_peered":"2024-09-10T20:27:17.282074+0000","last_unstale":"2024-09-10T20:33:29.829272+0000","last_undegraded":"2024-09-10T20:33:29.829272+0000","last_fullsized":"2024-09-10T20:33:29.829272+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:54:51.986119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.874322+0000","last_change":"2024-09-10T20:27:17.211220+0000","last_active":"2024-09-10T20:33:29.874322+0000","last_peered":"2024-09-10T20:33:29.874322+0000","last_clean":"2024-09-10T20:33:29.874322+0000","last_became_active":"2024-09-10T20:27:17.210803+0000","last_became_peered":"2024-09-10T20:27:17.210803+0000","last_unstale":"2024-09-10T20:33:29.874322+0000","last_undegraded":"2024-09-10T20:33:29.874322+0000","last_fullsized":"2024-09-10T20:33:29.874322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:48:38.315319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":579,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:33:29.876046+0000","last_change":"2024-09-10T20:27:17.311360+0000","last_active":"2024-09-10T20:33:29.876046+0000","last_peered":"2024-09-10T20:33:29.876046+0000","last_clean":"2024-09-10T20:33:29.876046+0000","last_became_active":"2024-09-10T20:27:17.310182+0000","last_became_peered":"2024-09-10T20:27:17.310182+0000","last_unstale":"2024-09-10T20:33:29.876046+0000","last_undegraded":"2024-09-10T20:33:29.876046+0000","last_fullsized":"2024-09-10T20:33:29.876046+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:27:16.136783+0000","last_clean_scrub_stamp":"2024-09-10T20:27:16.136783+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:34:02.949919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574957,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80416,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291424,"statfs":{"total":96636764160,"available":96554418176,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574956,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80488,"kb_used_data":4872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291352,"statfs":{"total":96636764160,"available":96554344448,"internally_reserved":0,"allocated":4988928,"data_stored":2922726,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574954,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76260,"kb_used_data":4812,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295580,"statfs":{"total":96636764160,"available":96558673920,"internally_reserved":0,"allocated":4927488,"data_stored":2393812,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640364,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72172,"kb_used_data":4868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299668,"statfs":{"total":96636764160,"available":96562860032,"internally_reserved":0,"allocated":4984832,"data_stored":2922721,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607660,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":67944,"kb_used_data":4816,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303896,"statfs":{"total":96636764160,"available":96567189504,"internally_reserved":0,"allocated":4931584,"data_stored":2393817,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":64616901},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607659,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72096,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299744,"statfs":{"total":96636764160,"available":96562937856,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:34:57.435 INFO:tasks.ceph:pgid 5.19 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.435 INFO:tasks.ceph:pgid 5.18 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.436 INFO:tasks.ceph:pgid 5.1b last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.436 INFO:tasks.ceph:pgid 5.1a last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.436 INFO:tasks.ceph:pgid 5.1d last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.436 INFO:tasks.ceph:pgid 5.1c last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.436 INFO:tasks.ceph:pgid 5.1f last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.436 INFO:tasks.ceph:pgid 5.1e last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.436 INFO:tasks.ceph:pgid 5.11 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.436 INFO:tasks.ceph:pgid 5.10 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.436 INFO:tasks.ceph:pgid 5.13 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.437 INFO:tasks.ceph:pgid 5.12 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.437 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-10T20:26:18.337408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=26, tm_sec=18, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.437 INFO:tasks.ceph:pgid 5.0 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.437 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-10T20:26:18.337408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=26, tm_sec=18, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.437 INFO:tasks.ceph:pgid 5.1 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.437 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-10T20:26:18.337408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=26, tm_sec=18, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.437 INFO:tasks.ceph:pgid 5.2 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.437 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-10T20:26:18.337408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=26, tm_sec=18, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.437 INFO:tasks.ceph:pgid 5.3 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.438 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-10T20:26:18.337408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=26, tm_sec=18, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.438 INFO:tasks.ceph:pgid 5.5 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.438 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-10T20:26:18.337408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=26, tm_sec=18, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.438 INFO:tasks.ceph:pgid 5.6 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.438 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-10T20:26:18.337408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=26, tm_sec=18, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.438 INFO:tasks.ceph:pgid 5.7 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.438 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-10T20:26:18.337408+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=26, tm_sec=18, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.438 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-10T20:26:13.287430+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=26, tm_sec=13, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.438 INFO:tasks.ceph:pgid 5.4 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.439 INFO:tasks.ceph:pgid 5.e last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.439 INFO:tasks.ceph:pgid 5.f last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.439 INFO:tasks.ceph:pgid 5.c last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.439 INFO:tasks.ceph:pgid 5.d last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.439 INFO:tasks.ceph:pgid 5.a last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.439 INFO:tasks.ceph:pgid 5.b last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.439 INFO:tasks.ceph:pgid 5.8 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.439 INFO:tasks.ceph:pgid 5.9 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.439 INFO:tasks.ceph:pgid 5.16 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.440 INFO:tasks.ceph:pgid 5.17 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.440 INFO:tasks.ceph:pgid 5.14 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.440 INFO:tasks.ceph:pgid 5.15 last_scrub_stamp 2024-09-10T20:27:16.136783+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=27, tm_sec=16, tm_wday=1, tm_yday=254, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=10, tm_hour=20, tm_min=34, tm_sec=53, tm_wday=1, tm_yday=254, tm_isdst=0) 2024-09-10T20:34:57.440 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-10T20:35:17.442 DEBUG:teuthology.orchestra.run.smithi153:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-10T20:35:17.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.569+0000 7fecef57c640 10 monclient: get_monmap_and_config 2024-09-10T20:35:17.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.569+0000 7fecef57c640 10 monclient: build_initial_monmap 2024-09-10T20:35:17.573 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.569+0000 7fecef57c640 10 monclient: monmap: 2024-09-10T20:35:17.573 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:35:17.573 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:35:17.572677+0000 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:35:17.572677+0000 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:35:17.574 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.569+0000 7fecef57c640 1 Processor -- start 2024-09-10T20:35:17.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.569+0000 7fecef57c640 1 -- start start 2024-09-10T20:35:17.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.569+0000 7fecef57c640 10 monclient: init 2024-09-10T20:35:17.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 10 monclient: _reopen_session rank -1 2024-09-10T20:35:17.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 10 monclient: _add_conns ranks=[4,2,0,3,1] 2024-09-10T20:35:17.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fece81364f0 0x7fece81368d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 10 monclient(hunting): picked mon.noname-c con 0x7fece81364f0 addr v2:172.21.15.176:3300/0 2024-09-10T20:35:17.577 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81360a0 0x7fece809ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 10 monclient(hunting): picked mon.noname-b con 0x7fece81360a0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:35:17.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fece80a1990 addr v1:172.21.15.153:6789/0 2024-09-10T20:35:17.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fece80a27a0 con 0x7fece80a1990 2024-09-10T20:35:17.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fece80a2a80 con 0x7fece81364f0 2024-09-10T20:35:17.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.578 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fece80a2d60 con 0x7fece81360a0 2024-09-10T20:35:17.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 10 monclient(hunting): _renew_subs 2024-09-10T20:35:17.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fecef57c640 10 monclient(hunting): authenticate will time out at 1477.712469s 2024-09-10T20:35:17.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81360a0 0x7fece809ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81360a0 0x7fece809ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35146/0 (socket says 172.21.15.153:35146) 2024-09-10T20:35:17.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 1 -- 172.21.15.153:0/339002069 learned_addr learned my addr 172.21.15.153:0/339002069 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:35:17.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient(hunting): get_auth_request con 0x7fece81360a0 auth_method 0 2024-09-10T20:35:17.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:35:17.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:35:17.579 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:35:17.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:35:17.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:35:17.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:35:17.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7feced318640 1 --2- 172.21.15.153:0/339002069 >> v2:172.21.15.176:3300/0 conn(0x7fece81364f0 0x7fece81368d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient(hunting): handle_auth_done global_id 54203 payload 274 2024-09-10T20:35:17.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 1 -- 172.21.15.153:0/339002069 >> v2:172.21.15.176:3300/0 conn(0x7fece81364f0 msgr2=0x7fece81368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:35:17.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 1 --2- 172.21.15.153:0/339002069 >> v2:172.21.15.176:3300/0 conn(0x7fece81364f0 0x7fece81368d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 1 -- 172.21.15.153:0/339002069 >> v1:172.21.15.153:6789/0 conn(0x7fece80a1990 legacy=0x7fece809d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:35:17.580 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient: _finish_hunting 0 2024-09-10T20:35:17.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 1 monclient: found mon.noname-b 2024-09-10T20:35:17.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:35:17.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.153:3301/0 2024-09-10T20:35:17.581 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.573+0000 7fececb17640 1 -- 172.21.15.153:0/339002069 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece8136e10 con 0x7fece81360a0 2024-09-10T20:35:17.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 1 -- 172.21.15.153:0/339002069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81360a0 msgr2=0x7fece809ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:35:17.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 1 --2- 172.21.15.153:0/339002069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81360a0 0x7fece809ce10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fece80a12f0 tx=0x7fecdc01c470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7b5262f2cead665 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:35:17.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 1 -- 172.21.15.153:0/339002069 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc0b0070 con 0x7fece81360a0 2024-09-10T20:35:17.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:35:17.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-10T20:35:17.588 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 10 monclient: dump: 2024-09-10T20:35:17.588 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:35:17.588 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 10 monclient: _finish_auth 0 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 10 monclient: _check_auth_tickets 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 5 monclient: authenticate success, global_id 54203 2024-09-10T20:35:17.589 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 1 -- 172.21.15.153:0/339002069 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecdc0ab600 con 0x7fece81360a0 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 10 monclient: handle_config config(0 keys) 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient: get_monmap_and_config success 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 1 -- 172.21.15.153:0/339002069 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecdc0ab9d0 con 0x7fece81360a0 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 4 set_mon_vals no callback set 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecdbfff640 10 monclient: dump: 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.590 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:35:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:35:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:35:17.591 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:35:17.591 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient: shutdown 2024-09-10T20:35:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:35:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 -- 172.21.15.153:0/339002069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81360a0 msgr2=0x7fece809ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 --2- 172.21.15.153:0/339002069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81360a0 0x7fece809ce10 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fece80a12f0 tx=0x7fecdc01c470 comp rx=0 tx=0).stop 2024-09-10T20:35:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 -- 172.21.15.153:0/339002069 shutdown_connections 2024-09-10T20:35:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 --2- 172.21.15.153:0/339002069 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81360a0 0x7fece809ce10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 --2- 172.21.15.153:0/339002069 >> v2:172.21.15.176:3300/0 conn(0x7fece81364f0 0x7fece81368d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.592 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 -- 172.21.15.153:0/339002069 >> 172.21.15.153:0/339002069 conn(0x7fece80a71b0 msgr2=0x7fece80a75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 -- 172.21.15.153:0/339002069 shutdown_connections 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 -- 172.21.15.153:0/339002069 wait complete. 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient: build_initial_monmap 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient: monmap: 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:35:17.593 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 librados: starting msgr at 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 librados: starting objecter 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 Processor -- start 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 -- start start 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 librados: setting wanted keys 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 librados: calling monclient init 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient: init 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient: _reopen_session rank -1 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fece80a1990 0x7fece8073c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient(hunting): picked mon.noname-b con 0x7fece80a1990 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:35:17.594 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece81360a0 0x7fece813d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient(hunting): picked mon.noname-a con 0x7fece81360a0 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:35:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81364f0 0x7fece81d0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient(hunting): picked mon.noname-c con 0x7fece81364f0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:35:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fece8140ff0 con 0x7fece81360a0 2024-09-10T20:35:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fece8140e70 con 0x7fece80a1990 2024-09-10T20:35:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.595 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fece8141170 con 0x7fece81364f0 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient(hunting): _renew_subs 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecef57c640 10 monclient(hunting): authenticate will time out at 1477.719619s 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece81360a0 0x7fece813d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecedb19640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81364f0 0x7fece81d0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece81360a0 0x7fece813d910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57210/0 (socket says 172.21.15.153:57210) 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 1 -- 172.21.15.153:0/2076824498 learned_addr learned my addr 172.21.15.153:0/2076824498 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecedb19640 10 monclient(hunting): get_auth_request con 0x7fece81364f0 auth_method 0 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecedb19640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecedb19640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecedb19640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:35:17.596 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 10 monclient(hunting): get_auth_request con 0x7fece81360a0 auth_method 0 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecedb19640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecedb19640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fecedb19640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7fececb17640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7feced318640 1 --2- 172.21.15.153:0/2076824498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fece80a1990 0x7fece8073c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7feced318640 10 monclient(hunting): get_auth_request con 0x7fece80a1990 auth_method 0 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7feced318640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:35:17.597 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7feced318640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.577+0000 7feced318640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 10 monclient(hunting): handle_auth_done global_id 54206 payload 824 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 1 -- 172.21.15.153:0/2076824498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fece80a1990 msgr2=0x7fece8073c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 1 --2- 172.21.15.153:0/2076824498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fece80a1990 0x7fece8073c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 1 -- 172.21.15.153:0/2076824498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece81360a0 msgr2=0x7fece813d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 1 --2- 172.21.15.153:0/2076824498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece81360a0 0x7fece813d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 10 monclient: _finish_hunting 0 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 1 monclient: found mon.noname-c 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:35:17.598 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:35:17.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 1 -- 172.21.15.153:0/2076824498 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fece80a69a0 con 0x7fece81364f0 2024-09-10T20:35:17.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fececb17640 1 --2- 172.21.15.153:0/2076824498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece81360a0 0x7fece813d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-09-10T20:35:17.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 1 -- 172.21.15.153:0/2076824498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81364f0 msgr2=0x7fece81d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:35:17.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecedb19640 1 --2- 172.21.15.153:0/2076824498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81364f0 0x7fece81d0950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fece000c780 tx=0x7fece000cc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:35:17.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 client.?.objecter ms_handle_connect 0x7fece81364f0 2024-09-10T20:35:17.599 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 client.?.objecter resend_mon_ops 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 1 -- 172.21.15.153:0/2076824498 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0016020 con 0x7fece81364f0 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 monclient: dump: 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:35:17.600 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 monclient: _finish_auth 0 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 monclient: _check_auth_tickets 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 5 monclient: authenticate success, global_id 54206 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 1 -- 172.21.15.153:0/2076824498 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fece000ac60 con 0x7fece81364f0 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 monclient: handle_config config(0 keys) 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 10 monclient: _renew_subs 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 1 -- 172.21.15.153:0/2076824498 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fece0012660 con 0x7fece81364f0 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:35:17.601 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fece81ce320 con 0x7fece81364f0 2024-09-10T20:35:17.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecdb7fe640 4 set_mon_vals no callback set 2024-09-10T20:35:17.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:35:17.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:35:17.602 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 10 monclient: dump: 2024-09-10T20:35:17.602 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:35:17.602 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.602 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.602 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 10 client.54206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 10 monclient: _renew_subs 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fece8074540 con 0x7fece81364f0 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 1 librados: init done 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:35:17.603 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fece81368d0 con 0x7fece81364f0 2024-09-10T20:35:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 1 -- 172.21.15.153:0/2076824498 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fece0019020 con 0x7fece81364f0 2024-09-10T20:35:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.581+0000 7fecd9ffb640 1 --2- 172.21.15.153:0/2076824498 >> v2:172.21.15.176:6812/3003473398 conn(0x7feccc03cef0 0x7feccc03f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.585+0000 7fecd9ffb640 1 -- 172.21.15.153:0/2076824498 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7fece004f4a0 con 0x7fece81364f0 2024-09-10T20:35:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.585+0000 7fecd9ffb640 10 client.54206.objecter ms_dispatch 0x7fece8000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:35:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.585+0000 7fecd9ffb640 3 client.54206.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:35:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.585+0000 7fecd9ffb640 3 client.54206.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:35:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.585+0000 7fecd9ffb640 20 client.54206.objecter dump_active .. 0 homeless 2024-09-10T20:35:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.585+0000 7feced318640 1 --2- 172.21.15.153:0/2076824498 >> v2:172.21.15.176:6812/3003473398 conn(0x7feccc03cef0 0x7feccc03f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.604 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.585+0000 7feced318640 10 monclient: get_auth_request con 0x7feccc03cef0 auth_method 0 2024-09-10T20:35:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.585+0000 7feced318640 1 --2- 172.21.15.153:0/2076824498 >> v2:172.21.15.176:6812/3003473398 conn(0x7feccc03cef0 0x7feccc03f3b0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fece400a970 tx=0x7fece4008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:35:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.585+0000 7fecd9ffb640 10 client.54206.objecter ms_handle_connect 0x7feccc03cef0 2024-09-10T20:35:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.589+0000 7fecd9ffb640 1 -- 172.21.15.153:0/2076824498 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fece0020030 con 0x7fece81364f0 2024-09-10T20:35:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.589+0000 7fecd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:35:17.605 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.589+0000 7fecd9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:35:17.752 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.749+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 --> v2:172.21.15.176:6812/3003473398 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fece806a640 con 0x7feccc03cef0 2024-09-10T20:35:17.758 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecd9ffb640 1 -- 172.21.15.153:0/2076824498 <== mgr.4106 v2:172.21.15.176:6812/3003473398 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114167 (secure 0 0 0) 0x7fece806a640 con 0x7feccc03cef0 2024-09-10T20:35:17.759 INFO:teuthology.orchestra.run.smithi153.stdout: 2024-09-10T20:35:17.760 INFO:teuthology.orchestra.run.smithi153.stderr:dumped all 2024-09-10T20:35:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 10 librados: watch_flush enter 2024-09-10T20:35:17.761 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 10 librados: watch_flush exit 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 20 client.54206.objecter shutdown clearing up homeless session... 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 10 client.54206.objecter successfully canceled tick 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 >> v2:172.21.15.176:6812/3003473398 conn(0x7feccc03cef0 msgr2=0x7feccc03f3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 --2- 172.21.15.153:0/2076824498 >> v2:172.21.15.176:6812/3003473398 conn(0x7feccc03cef0 0x7feccc03f3b0 secure :-1 s=READY pgs=3389 cs=0 l=1 rev1=1 crypto rx=0x7fece400a970 tx=0x7fece4008040 comp rx=0 tx=0).stop 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 10 monclient: shutdown 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81364f0 msgr2=0x7fece81d0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 --2- 172.21.15.153:0/2076824498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81364f0 0x7fece81d0950 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fece000c780 tx=0x7fece000cc40 comp rx=0 tx=0).stop 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 shutdown_connections 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 --2- 172.21.15.153:0/2076824498 >> v2:172.21.15.176:6812/3003473398 conn(0x7feccc03cef0 0x7feccc03f3b0 unknown :-1 s=CLOSED pgs=3389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 --2- 172.21.15.153:0/2076824498 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fece81364f0 0x7fece81d0950 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.762 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 --2- 172.21.15.153:0/2076824498 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fece81360a0 0x7fece813d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 --2- 172.21.15.153:0/2076824498 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fece80a1990 0x7fece8073c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 >> 172.21.15.153:0/2076824498 conn(0x7fece80a71b0 msgr2=0x7fece812c800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:35:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 shutdown_connections 2024-09-10T20:35:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 -- 172.21.15.153:0/2076824498 wait complete. 2024-09-10T20:35:17.763 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.757+0000 7fecef57c640 1 librados: shutdown 2024-09-10T20:35:17.776 INFO:teuthology.orchestra.run.smithi153.stdout:{"pg_ready":true,"pg_map":{"version":584,"stamp":"2024-09-10T20:35:17.257773+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":453464,"kb_used_data":28984,"kb_used_omap":9,"kb_used_meta":424374,"kb_avail":565777576,"statfs":{"total":579820584960,"available":579356237824,"internally_reserved":0,"allocated":29679616,"data_stored":15420662,"data_compressed":596482,"data_compressed_allocated":966656,"data_compressed_original":3112960,"omap_allocated":9567,"internal_metadata":434559649},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003854"},"pg_stats":[{"pgid":"5.19","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:02.003212+0000","last_change":"2024-09-10T20:35:02.003212+0000","last_active":"2024-09-10T20:35:02.003212+0000","last_peered":"2024-09-10T20:35:02.003212+0000","last_clean":"2024-09-10T20:35:02.003212+0000","last_became_active":"2024-09-10T20:27:17.197839+0000","last_became_peered":"2024-09-10T20:27:17.197839+0000","last_unstale":"2024-09-10T20:35:02.003212+0000","last_undegraded":"2024-09-10T20:35:02.003212+0000","last_fullsized":"2024-09-10T20:35:02.003212+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:02.003052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:02.003052+0000","last_clean_scrub_stamp":"2024-09-10T20:35:02.003052+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:52.742743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":589,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:03.976121+0000","last_change":"2024-09-10T20:35:03.976121+0000","last_active":"2024-09-10T20:35:03.976121+0000","last_peered":"2024-09-10T20:35:03.976121+0000","last_clean":"2024-09-10T20:35:03.976121+0000","last_became_active":"2024-09-10T20:27:17.219498+0000","last_became_peered":"2024-09-10T20:27:17.219498+0000","last_unstale":"2024-09-10T20:35:03.976121+0000","last_undegraded":"2024-09-10T20:35:03.976121+0000","last_fullsized":"2024-09-10T20:35:03.976121+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:03.975955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:03.975955+0000","last_clean_scrub_stamp":"2024-09-10T20:35:03.975955+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:12:57.665627+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":602,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:00.099501+0000","last_change":"2024-09-10T20:35:00.099501+0000","last_active":"2024-09-10T20:35:00.099501+0000","last_peered":"2024-09-10T20:35:00.099501+0000","last_clean":"2024-09-10T20:35:00.099501+0000","last_became_active":"2024-09-10T20:27:17.202373+0000","last_became_peered":"2024-09-10T20:27:17.202373+0000","last_unstale":"2024-09-10T20:35:00.099501+0000","last_undegraded":"2024-09-10T20:35:00.099501+0000","last_fullsized":"2024-09-10T20:35:00.099501+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:00.099374+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:00.099374+0000","last_clean_scrub_stamp":"2024-09-10T20:35:00.099374+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:41:16.001486+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":589,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:05.963478+0000","last_change":"2024-09-10T20:35:05.963478+0000","last_active":"2024-09-10T20:35:05.963478+0000","last_peered":"2024-09-10T20:35:05.963478+0000","last_clean":"2024-09-10T20:35:05.963478+0000","last_became_active":"2024-09-10T20:27:17.217517+0000","last_became_peered":"2024-09-10T20:27:17.217517+0000","last_unstale":"2024-09-10T20:35:05.963478+0000","last_undegraded":"2024-09-10T20:35:05.963478+0000","last_fullsized":"2024-09-10T20:35:05.963478+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:05.963308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:05.963308+0000","last_clean_scrub_stamp":"2024-09-10T20:35:05.963308+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:17:52.574479+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":586,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:02.980879+0000","last_change":"2024-09-10T20:35:02.980879+0000","last_active":"2024-09-10T20:35:02.980879+0000","last_peered":"2024-09-10T20:35:02.980879+0000","last_clean":"2024-09-10T20:35:02.980879+0000","last_became_active":"2024-09-10T20:27:17.281836+0000","last_became_peered":"2024-09-10T20:27:17.281836+0000","last_unstale":"2024-09-10T20:35:02.980879+0000","last_undegraded":"2024-09-10T20:35:02.980879+0000","last_fullsized":"2024-09-10T20:35:02.980879+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:02.980721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:02.980721+0000","last_clean_scrub_stamp":"2024-09-10T20:35:02.980721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:04.712235+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":602,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:59.064514+0000","last_change":"2024-09-10T20:34:59.064514+0000","last_active":"2024-09-10T20:34:59.064514+0000","last_peered":"2024-09-10T20:34:59.064514+0000","last_clean":"2024-09-10T20:34:59.064514+0000","last_became_active":"2024-09-10T20:27:17.212620+0000","last_became_peered":"2024-09-10T20:27:17.212620+0000","last_unstale":"2024-09-10T20:34:59.064514+0000","last_undegraded":"2024-09-10T20:34:59.064514+0000","last_fullsized":"2024-09-10T20:34:59.064514+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:59.064362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:59.064362+0000","last_clean_scrub_stamp":"2024-09-10T20:34:59.064362+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:35:08.468590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":602,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:01.142197+0000","last_change":"2024-09-10T20:35:01.142197+0000","last_active":"2024-09-10T20:35:01.142197+0000","last_peered":"2024-09-10T20:35:01.142197+0000","last_clean":"2024-09-10T20:35:01.142197+0000","last_became_active":"2024-09-10T20:27:17.287973+0000","last_became_peered":"2024-09-10T20:27:17.287973+0000","last_unstale":"2024-09-10T20:35:01.142197+0000","last_undegraded":"2024-09-10T20:35:01.142197+0000","last_fullsized":"2024-09-10T20:35:01.142197+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:01.142034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:01.142034+0000","last_clean_scrub_stamp":"2024-09-10T20:35:01.142034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T06:53:20.455226+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:02.027010+0000","last_change":"2024-09-10T20:35:02.027010+0000","last_active":"2024-09-10T20:35:02.027010+0000","last_peered":"2024-09-10T20:35:02.027010+0000","last_clean":"2024-09-10T20:35:02.027010+0000","last_became_active":"2024-09-10T20:27:17.209693+0000","last_became_peered":"2024-09-10T20:27:17.209693+0000","last_unstale":"2024-09-10T20:35:02.027010+0000","last_undegraded":"2024-09-10T20:35:02.027010+0000","last_fullsized":"2024-09-10T20:35:02.027010+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:02.026858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:02.026858+0000","last_clean_scrub_stamp":"2024-09-10T20:35:02.026858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:04:12.133669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":589,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:02.997567+0000","last_change":"2024-09-10T20:35:02.997567+0000","last_active":"2024-09-10T20:35:02.997567+0000","last_peered":"2024-09-10T20:35:02.997567+0000","last_clean":"2024-09-10T20:35:02.997567+0000","last_became_active":"2024-09-10T20:27:17.249683+0000","last_became_peered":"2024-09-10T20:27:17.249683+0000","last_unstale":"2024-09-10T20:35:02.997567+0000","last_undegraded":"2024-09-10T20:35:02.997567+0000","last_fullsized":"2024-09-10T20:35:02.997567+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:02.997405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:02.997405+0000","last_clean_scrub_stamp":"2024-09-10T20:35:02.997405+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:00:28.284302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":602,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:58.067957+0000","last_change":"2024-09-10T20:34:58.067957+0000","last_active":"2024-09-10T20:34:58.067957+0000","last_peered":"2024-09-10T20:34:58.067957+0000","last_clean":"2024-09-10T20:34:58.067957+0000","last_became_active":"2024-09-10T20:27:17.247865+0000","last_became_peered":"2024-09-10T20:27:17.247865+0000","last_unstale":"2024-09-10T20:34:58.067957+0000","last_undegraded":"2024-09-10T20:34:58.067957+0000","last_fullsized":"2024-09-10T20:34:58.067957+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:58.067785+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:58.067785+0000","last_clean_scrub_stamp":"2024-09-10T20:34:58.067785+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:43:48.169529+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":586,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:00.035358+0000","last_change":"2024-09-10T20:35:00.035358+0000","last_active":"2024-09-10T20:35:00.035358+0000","last_peered":"2024-09-10T20:35:00.035358+0000","last_clean":"2024-09-10T20:35:00.035358+0000","last_became_active":"2024-09-10T20:27:17.203002+0000","last_became_peered":"2024-09-10T20:27:17.203002+0000","last_unstale":"2024-09-10T20:35:00.035358+0000","last_undegraded":"2024-09-10T20:35:00.035358+0000","last_fullsized":"2024-09-10T20:35:00.035358+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:00.035168+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:00.035168+0000","last_clean_scrub_stamp":"2024-09-10T20:35:00.035168+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:21:58.931403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:00.839183+0000","last_change":"2024-09-10T20:35:00.839183+0000","last_active":"2024-09-10T20:35:00.839183+0000","last_peered":"2024-09-10T20:35:00.839183+0000","last_clean":"2024-09-10T20:35:00.839183+0000","last_became_active":"2024-09-10T20:27:17.298309+0000","last_became_peered":"2024-09-10T20:27:17.298309+0000","last_unstale":"2024-09-10T20:35:00.839183+0000","last_undegraded":"2024-09-10T20:35:00.839183+0000","last_fullsized":"2024-09-10T20:35:00.839183+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:00.839016+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:00.839016+0000","last_clean_scrub_stamp":"2024-09-10T20:35:00.839016+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:11:03.229822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":605,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:58.961549+0000","last_change":"2024-09-10T20:34:58.961408+0000","last_active":"2024-09-10T20:34:58.961549+0000","last_peered":"2024-09-10T20:34:58.961549+0000","last_clean":"2024-09-10T20:34:58.961549+0000","last_became_active":"2024-09-10T20:26:19.356844+0000","last_became_peered":"2024-09-10T20:26:19.356844+0000","last_unstale":"2024-09-10T20:34:58.961549+0000","last_undegraded":"2024-09-10T20:34:58.961549+0000","last_fullsized":"2024-09-10T20:34:58.961549+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:58.961239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:58.961239+0000","last_clean_scrub_stamp":"2024-09-10T20:34:58.961239+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:53:00.559492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0007561,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:57.871484+0000","last_change":"2024-09-10T20:34:57.871484+0000","last_active":"2024-09-10T20:34:57.871484+0000","last_peered":"2024-09-10T20:34:57.871484+0000","last_clean":"2024-09-10T20:34:57.871484+0000","last_became_active":"2024-09-10T20:27:17.205464+0000","last_became_peered":"2024-09-10T20:27:17.205464+0000","last_unstale":"2024-09-10T20:34:57.871484+0000","last_undegraded":"2024-09-10T20:34:57.871484+0000","last_fullsized":"2024-09-10T20:34:57.871484+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:57.871323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:57.871323+0000","last_clean_scrub_stamp":"2024-09-10T20:34:57.871323+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:36:34.578082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":602,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:57.031742+0000","last_change":"2024-09-10T20:34:57.031498+0000","last_active":"2024-09-10T20:34:57.031742+0000","last_peered":"2024-09-10T20:34:57.031742+0000","last_clean":"2024-09-10T20:34:57.031742+0000","last_became_active":"2024-09-10T20:26:19.363220+0000","last_became_peered":"2024-09-10T20:26:19.363220+0000","last_unstale":"2024-09-10T20:34:57.031742+0000","last_undegraded":"2024-09-10T20:34:57.031742+0000","last_fullsized":"2024-09-10T20:34:57.031742+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:57.031306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:57.031306+0000","last_clean_scrub_stamp":"2024-09-10T20:34:57.031306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:39:19.686089+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.00096679600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"317'8","reported_seq":603,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:59.925886+0000","last_change":"2024-09-10T20:34:59.925886+0000","last_active":"2024-09-10T20:34:59.925886+0000","last_peered":"2024-09-10T20:34:59.925886+0000","last_clean":"2024-09-10T20:34:59.925886+0000","last_became_active":"2024-09-10T20:27:17.211037+0000","last_became_peered":"2024-09-10T20:27:17.211037+0000","last_unstale":"2024-09-10T20:34:59.925886+0000","last_undegraded":"2024-09-10T20:34:59.925886+0000","last_fullsized":"2024-09-10T20:34:59.925886+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'8","last_scrub_stamp":"2024-09-10T20:34:59.925701+0000","last_deep_scrub":"317'8","last_deep_scrub_stamp":"2024-09-10T20:34:59.925701+0000","last_clean_scrub_stamp":"2024-09-10T20:34:59.925701+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:34:31.929373+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":605,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:57.924607+0000","last_change":"2024-09-10T20:34:57.924607+0000","last_active":"2024-09-10T20:34:57.924607+0000","last_peered":"2024-09-10T20:34:57.924607+0000","last_clean":"2024-09-10T20:34:57.924607+0000","last_became_active":"2024-09-10T20:26:19.363294+0000","last_became_peered":"2024-09-10T20:26:19.363294+0000","last_unstale":"2024-09-10T20:34:57.924607+0000","last_undegraded":"2024-09-10T20:34:57.924607+0000","last_fullsized":"2024-09-10T20:34:57.924607+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:57.924451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:57.924451+0000","last_clean_scrub_stamp":"2024-09-10T20:34:57.924451+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:20:13.598947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053952999999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:56.791942+0000","last_change":"2024-09-10T20:34:56.791942+0000","last_active":"2024-09-10T20:34:56.791942+0000","last_peered":"2024-09-10T20:34:56.791942+0000","last_clean":"2024-09-10T20:34:56.791942+0000","last_became_active":"2024-09-10T20:27:17.297745+0000","last_became_peered":"2024-09-10T20:27:17.297745+0000","last_unstale":"2024-09-10T20:34:56.791942+0000","last_undegraded":"2024-09-10T20:34:56.791942+0000","last_fullsized":"2024-09-10T20:34:56.791942+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:56.791859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:56.791859+0000","last_clean_scrub_stamp":"2024-09-10T20:34:56.791859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T07:25:27.004778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":603,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:59.051206+0000","last_change":"2024-09-10T20:34:59.051206+0000","last_active":"2024-09-10T20:34:59.051206+0000","last_peered":"2024-09-10T20:34:59.051206+0000","last_clean":"2024-09-10T20:34:59.051206+0000","last_became_active":"2024-09-10T20:26:19.359961+0000","last_became_peered":"2024-09-10T20:26:19.359961+0000","last_unstale":"2024-09-10T20:34:59.051206+0000","last_undegraded":"2024-09-10T20:34:59.051206+0000","last_fullsized":"2024-09-10T20:34:59.051206+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:59.051034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:59.051034+0000","last_clean_scrub_stamp":"2024-09-10T20:34:59.051034+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:24:37.270529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001234743,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":586,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:58.061781+0000","last_change":"2024-09-10T20:34:58.061781+0000","last_active":"2024-09-10T20:34:58.061781+0000","last_peered":"2024-09-10T20:34:58.061781+0000","last_clean":"2024-09-10T20:34:58.061781+0000","last_became_active":"2024-09-10T20:27:17.281815+0000","last_became_peered":"2024-09-10T20:27:17.281815+0000","last_unstale":"2024-09-10T20:34:58.061781+0000","last_undegraded":"2024-09-10T20:34:58.061781+0000","last_fullsized":"2024-09-10T20:34:58.061781+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:58.061624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:58.061624+0000","last_clean_scrub_stamp":"2024-09-10T20:34:58.061624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:19:03.048281+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"317'3","reported_seq":606,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:55.779788+0000","last_change":"2024-09-10T20:34:55.779788+0000","last_active":"2024-09-10T20:34:55.779788+0000","last_peered":"2024-09-10T20:34:55.779788+0000","last_clean":"2024-09-10T20:34:55.779788+0000","last_became_active":"2024-09-10T20:26:19.360427+0000","last_became_peered":"2024-09-10T20:26:19.360427+0000","last_unstale":"2024-09-10T20:34:55.779788+0000","last_undegraded":"2024-09-10T20:34:55.779788+0000","last_fullsized":"2024-09-10T20:34:55.779788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'3","last_scrub_stamp":"2024-09-10T20:34:55.779727+0000","last_deep_scrub":"317'3","last_deep_scrub_stamp":"2024-09-10T20:34:55.779727+0000","last_clean_scrub_stamp":"2024-09-10T20:34:55.779727+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:24:34.534161+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0020581079999999999,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:58.062954+0000","last_change":"2024-09-10T20:34:58.062954+0000","last_active":"2024-09-10T20:34:58.062954+0000","last_peered":"2024-09-10T20:34:58.062954+0000","last_clean":"2024-09-10T20:34:58.062954+0000","last_became_active":"2024-09-10T20:27:17.282600+0000","last_became_peered":"2024-09-10T20:27:17.282600+0000","last_unstale":"2024-09-10T20:34:58.062954+0000","last_undegraded":"2024-09-10T20:34:58.062954+0000","last_fullsized":"2024-09-10T20:34:58.062954+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:58.062797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:58.062797+0000","last_clean_scrub_stamp":"2024-09-10T20:34:58.062797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:35:42.701681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":611,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:56.904147+0000","last_change":"2024-09-10T20:34:56.904147+0000","last_active":"2024-09-10T20:34:56.904147+0000","last_peered":"2024-09-10T20:34:56.904147+0000","last_clean":"2024-09-10T20:34:56.904147+0000","last_became_active":"2024-09-10T20:26:19.362858+0000","last_became_peered":"2024-09-10T20:26:19.362858+0000","last_unstale":"2024-09-10T20:34:56.904147+0000","last_undegraded":"2024-09-10T20:34:56.904147+0000","last_fullsized":"2024-09-10T20:34:56.904147+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:56.903957+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:56.903957+0000","last_clean_scrub_stamp":"2024-09-10T20:34:56.903957+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:38:07.025807+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00078714099999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":586,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:59.028888+0000","last_change":"2024-09-10T20:34:59.028888+0000","last_active":"2024-09-10T20:34:59.028888+0000","last_peered":"2024-09-10T20:34:59.028888+0000","last_clean":"2024-09-10T20:34:59.028888+0000","last_became_active":"2024-09-10T20:27:17.214428+0000","last_became_peered":"2024-09-10T20:27:17.214428+0000","last_unstale":"2024-09-10T20:34:59.028888+0000","last_undegraded":"2024-09-10T20:34:59.028888+0000","last_fullsized":"2024-09-10T20:34:59.028888+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:59.028724+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:59.028724+0000","last_clean_scrub_stamp":"2024-09-10T20:34:59.028724+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T21:22:37.167699+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":618,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:56.039374+0000","last_change":"2024-09-10T20:34:56.039374+0000","last_active":"2024-09-10T20:34:56.039374+0000","last_peered":"2024-09-10T20:34:56.039374+0000","last_clean":"2024-09-10T20:34:56.039374+0000","last_became_active":"2024-09-10T20:26:19.362158+0000","last_became_peered":"2024-09-10T20:26:19.362158+0000","last_unstale":"2024-09-10T20:34:56.039374+0000","last_undegraded":"2024-09-10T20:34:56.039374+0000","last_fullsized":"2024-09-10T20:34:56.039374+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:56.039309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:56.039309+0000","last_clean_scrub_stamp":"2024-09-10T20:34:56.039309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:00.047748+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010122729999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:57.773157+0000","last_change":"2024-09-10T20:34:57.773157+0000","last_active":"2024-09-10T20:34:57.773157+0000","last_peered":"2024-09-10T20:34:57.773157+0000","last_clean":"2024-09-10T20:34:57.773157+0000","last_became_active":"2024-09-10T20:27:17.243935+0000","last_became_peered":"2024-09-10T20:27:17.243935+0000","last_unstale":"2024-09-10T20:34:57.773157+0000","last_undegraded":"2024-09-10T20:34:57.773157+0000","last_fullsized":"2024-09-10T20:34:57.773157+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:57.772993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:57.772993+0000","last_clean_scrub_stamp":"2024-09-10T20:34:57.772993+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:28:04.218822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.3","version":"317'2","reported_seq":607,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:56.901546+0000","last_change":"2024-09-10T20:34:56.901546+0000","last_active":"2024-09-10T20:34:56.901546+0000","last_peered":"2024-09-10T20:34:56.901546+0000","last_clean":"2024-09-10T20:34:56.901546+0000","last_became_active":"2024-09-10T20:26:19.359775+0000","last_became_peered":"2024-09-10T20:26:19.359775+0000","last_unstale":"2024-09-10T20:34:56.901546+0000","last_undegraded":"2024-09-10T20:34:56.901546+0000","last_fullsized":"2024-09-10T20:34:56.901546+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'2","last_scrub_stamp":"2024-09-10T20:34:56.901487+0000","last_deep_scrub":"317'2","last_deep_scrub_stamp":"2024-09-10T20:34:56.901487+0000","last_clean_scrub_stamp":"2024-09-10T20:34:56.901487+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:56:34.787477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054594699999999997,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"317'74","reported_seq":698,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:55.868135+0000","last_change":"2024-09-10T20:34:55.868135+0000","last_active":"2024-09-10T20:34:55.868135+0000","last_peered":"2024-09-10T20:34:55.868135+0000","last_clean":"2024-09-10T20:34:55.868135+0000","last_became_active":"2024-09-10T20:26:14.577602+0000","last_became_peered":"2024-09-10T20:26:14.577602+0000","last_unstale":"2024-09-10T20:34:55.868135+0000","last_undegraded":"2024-09-10T20:34:55.868135+0000","last_fullsized":"2024-09-10T20:34:55.868135+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'74","last_scrub_stamp":"2024-09-10T20:34:55.868065+0000","last_deep_scrub":"317'74","last_deep_scrub_stamp":"2024-09-10T20:34:55.868065+0000","last_clean_scrub_stamp":"2024-09-10T20:34:55.868065+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:18:11.403154+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:54.788789+0000","last_change":"2024-09-10T20:34:54.788789+0000","last_active":"2024-09-10T20:34:54.788789+0000","last_peered":"2024-09-10T20:34:54.788789+0000","last_clean":"2024-09-10T20:34:54.788789+0000","last_became_active":"2024-09-10T20:27:17.295709+0000","last_became_peered":"2024-09-10T20:27:17.295709+0000","last_unstale":"2024-09-10T20:34:54.788789+0000","last_undegraded":"2024-09-10T20:34:54.788789+0000","last_fullsized":"2024-09-10T20:34:54.788789+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:54.788627+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:54.788627+0000","last_clean_scrub_stamp":"2024-09-10T20:34:54.788627+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:02:57.364784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":602,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:57.062947+0000","last_change":"2024-09-10T20:34:57.062947+0000","last_active":"2024-09-10T20:34:57.062947+0000","last_peered":"2024-09-10T20:34:57.062947+0000","last_clean":"2024-09-10T20:34:57.062947+0000","last_became_active":"2024-09-10T20:27:17.217973+0000","last_became_peered":"2024-09-10T20:27:17.217973+0000","last_unstale":"2024-09-10T20:34:57.062947+0000","last_undegraded":"2024-09-10T20:34:57.062947+0000","last_fullsized":"2024-09-10T20:34:57.062947+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:57.062732+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:57.062732+0000","last_clean_scrub_stamp":"2024-09-10T20:34:57.062732+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:33:37.907998+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":589,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:00.950559+0000","last_change":"2024-09-10T20:35:00.950559+0000","last_active":"2024-09-10T20:35:00.950559+0000","last_peered":"2024-09-10T20:35:00.950559+0000","last_clean":"2024-09-10T20:35:00.950559+0000","last_became_active":"2024-09-10T20:27:17.318202+0000","last_became_peered":"2024-09-10T20:27:17.318202+0000","last_unstale":"2024-09-10T20:35:00.950559+0000","last_undegraded":"2024-09-10T20:35:00.950559+0000","last_fullsized":"2024-09-10T20:35:00.950559+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:00.950398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:00.950398+0000","last_clean_scrub_stamp":"2024-09-10T20:35:00.950398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T03:04:39.054343+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:03.001608+0000","last_change":"2024-09-10T20:35:03.001608+0000","last_active":"2024-09-10T20:35:03.001608+0000","last_peered":"2024-09-10T20:35:03.001608+0000","last_clean":"2024-09-10T20:35:03.001608+0000","last_became_active":"2024-09-10T20:27:17.240435+0000","last_became_peered":"2024-09-10T20:27:17.240435+0000","last_unstale":"2024-09-10T20:35:03.001608+0000","last_undegraded":"2024-09-10T20:35:03.001608+0000","last_fullsized":"2024-09-10T20:35:03.001608+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:03.001446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:03.001446+0000","last_clean_scrub_stamp":"2024-09-10T20:35:03.001446+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:56:00.215624+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:59.879666+0000","last_change":"2024-09-10T20:34:59.879666+0000","last_active":"2024-09-10T20:34:59.879666+0000","last_peered":"2024-09-10T20:34:59.879666+0000","last_clean":"2024-09-10T20:34:59.879666+0000","last_became_active":"2024-09-10T20:27:17.229886+0000","last_became_peered":"2024-09-10T20:27:17.229886+0000","last_unstale":"2024-09-10T20:34:59.879666+0000","last_undegraded":"2024-09-10T20:34:59.879666+0000","last_fullsized":"2024-09-10T20:34:59.879666+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:59.879500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:59.879500+0000","last_clean_scrub_stamp":"2024-09-10T20:34:59.879500+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:08:50.537101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:58.870944+0000","last_change":"2024-09-10T20:34:58.870944+0000","last_active":"2024-09-10T20:34:58.870944+0000","last_peered":"2024-09-10T20:34:58.870944+0000","last_clean":"2024-09-10T20:34:58.870944+0000","last_became_active":"2024-09-10T20:27:17.213875+0000","last_became_peered":"2024-09-10T20:27:17.213875+0000","last_unstale":"2024-09-10T20:34:58.870944+0000","last_undegraded":"2024-09-10T20:34:58.870944+0000","last_fullsized":"2024-09-10T20:34:58.870944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:58.870788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:58.870788+0000","last_clean_scrub_stamp":"2024-09-10T20:34:58.870788+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T04:10:03.925513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:00.995287+0000","last_change":"2024-09-10T20:35:00.995287+0000","last_active":"2024-09-10T20:35:00.995287+0000","last_peered":"2024-09-10T20:35:00.995287+0000","last_clean":"2024-09-10T20:35:00.995287+0000","last_became_active":"2024-09-10T20:27:17.288092+0000","last_became_peered":"2024-09-10T20:27:17.288092+0000","last_unstale":"2024-09-10T20:35:00.995287+0000","last_undegraded":"2024-09-10T20:35:00.995287+0000","last_fullsized":"2024-09-10T20:35:00.995287+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:00.995114+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:00.995114+0000","last_clean_scrub_stamp":"2024-09-10T20:35:00.995114+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T05:05:59.768766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:00.019606+0000","last_change":"2024-09-10T20:35:00.019606+0000","last_active":"2024-09-10T20:35:00.019606+0000","last_peered":"2024-09-10T20:35:00.019606+0000","last_clean":"2024-09-10T20:35:00.019606+0000","last_became_active":"2024-09-10T20:27:17.209363+0000","last_became_peered":"2024-09-10T20:27:17.209363+0000","last_unstale":"2024-09-10T20:35:00.019606+0000","last_undegraded":"2024-09-10T20:35:00.019606+0000","last_fullsized":"2024-09-10T20:35:00.019606+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:00.019439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:00.019439+0000","last_clean_scrub_stamp":"2024-09-10T20:35:00.019439+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T01:51:27.413804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":589,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:01.952139+0000","last_change":"2024-09-10T20:35:01.952139+0000","last_active":"2024-09-10T20:35:01.952139+0000","last_peered":"2024-09-10T20:35:01.952139+0000","last_clean":"2024-09-10T20:35:01.952139+0000","last_became_active":"2024-09-10T20:27:17.224230+0000","last_became_peered":"2024-09-10T20:27:17.224230+0000","last_unstale":"2024-09-10T20:35:01.952139+0000","last_undegraded":"2024-09-10T20:35:01.952139+0000","last_fullsized":"2024-09-10T20:35:01.952139+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:01.951975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:01.951975+0000","last_clean_scrub_stamp":"2024-09-10T20:35:01.951975+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T22:37:23.778681+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":589,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:04.974703+0000","last_change":"2024-09-10T20:35:04.974703+0000","last_active":"2024-09-10T20:35:04.974703+0000","last_peered":"2024-09-10T20:35:04.974703+0000","last_clean":"2024-09-10T20:35:04.974703+0000","last_became_active":"2024-09-10T20:27:17.217107+0000","last_became_peered":"2024-09-10T20:27:17.217107+0000","last_unstale":"2024-09-10T20:35:04.974703+0000","last_undegraded":"2024-09-10T20:35:04.974703+0000","last_fullsized":"2024-09-10T20:35:04.974703+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:04.974538+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:04.974538+0000","last_clean_scrub_stamp":"2024-09-10T20:35:04.974538+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:23:23.045123+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":586,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:01.034398+0000","last_change":"2024-09-10T20:35:01.034398+0000","last_active":"2024-09-10T20:35:01.034398+0000","last_peered":"2024-09-10T20:35:01.034398+0000","last_clean":"2024-09-10T20:35:01.034398+0000","last_became_active":"2024-09-10T20:27:17.282074+0000","last_became_peered":"2024-09-10T20:27:17.282074+0000","last_unstale":"2024-09-10T20:35:01.034398+0000","last_undegraded":"2024-09-10T20:35:01.034398+0000","last_fullsized":"2024-09-10T20:35:01.034398+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:01.034237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:01.034237+0000","last_clean_scrub_stamp":"2024-09-10T20:35:01.034237+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T00:41:10.370710+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":595,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:35:01.851123+0000","last_change":"2024-09-10T20:35:01.851123+0000","last_active":"2024-09-10T20:35:01.851123+0000","last_peered":"2024-09-10T20:35:01.851123+0000","last_clean":"2024-09-10T20:35:01.851123+0000","last_became_active":"2024-09-10T20:27:17.210803+0000","last_became_peered":"2024-09-10T20:27:17.210803+0000","last_unstale":"2024-09-10T20:35:01.851123+0000","last_undegraded":"2024-09-10T20:35:01.851123+0000","last_fullsized":"2024-09-10T20:35:01.851123+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:35:01.850963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:35:01.850963+0000","last_clean_scrub_stamp":"2024-09-10T20:35:01.850963+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T02:19:29.312297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":587,"reported_epoch":317,"state":"active+clean","last_fresh":"2024-09-10T20:34:58.799441+0000","last_change":"2024-09-10T20:34:58.799441+0000","last_active":"2024-09-10T20:34:58.799441+0000","last_peered":"2024-09-10T20:34:58.799441+0000","last_clean":"2024-09-10T20:34:58.799441+0000","last_became_active":"2024-09-10T20:27:17.310182+0000","last_became_peered":"2024-09-10T20:27:17.310182+0000","last_unstale":"2024-09-10T20:34:58.799441+0000","last_undegraded":"2024-09-10T20:34:58.799441+0000","last_fullsized":"2024-09-10T20:34:58.799441+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-10T20:34:58.799241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-10T20:34:58.799241+0000","last_clean_scrub_stamp":"2024-09-10T20:34:58.799241+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-11T23:00:34.246976+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":60928,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":13,"seq":55834574961,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80408,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291432,"statfs":{"total":96636764160,"available":96554426368,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574960,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80472,"kb_used_data":4872,"kb_used_omap":1,"kb_used_meta":75582,"kb_avail":94291368,"statfs":{"total":96636764160,"available":96554360832,"internally_reserved":0,"allocated":4988928,"data_stored":2922726,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":77396421},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574958,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":76252,"kb_used_data":4812,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":94295588,"statfs":{"total":96636764160,"available":96558682112,"internally_reserved":0,"allocated":4927488,"data_stored":2393812,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":11,"seq":47244640368,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72148,"kb_used_data":4868,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299692,"statfs":{"total":96636764160,"available":96562884608,"internally_reserved":0,"allocated":4984832,"data_stored":2922721,"data_compressed":119723,"data_compressed_allocated":188416,"data_compressed_original":860160,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607664,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72096,"kb_used_data":4816,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299744,"statfs":{"total":96636764160,"available":96562937856,"internally_reserved":0,"allocated":4931584,"data_stored":2393817,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":12,"seq":51539607663,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72088,"kb_used_data":4808,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299752,"statfs":{"total":96636764160,"available":96562946048,"internally_reserved":0,"allocated":4923392,"data_stored":2393793,"data_compressed":89259,"data_compressed_allocated":147456,"data_compressed_original":348160,"omap_allocated":1592,"internal_metadata":68876744},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30464,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-10T20:35:17.780 DEBUG:teuthology.orchestra.run.smithi153:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-10T20:35:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.885+0000 7fda28a00640 10 monclient: get_monmap_and_config 2024-09-10T20:35:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.885+0000 7fda28a00640 10 monclient: build_initial_monmap 2024-09-10T20:35:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.885+0000 7fda28a00640 10 monclient: monmap: 2024-09-10T20:35:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:35:17.889 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:35:17.889187+0000 2024-09-10T20:35:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:35:17.889187+0000 2024-09-10T20:35:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:35:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:0: v1:172.21.15.153:6789/0 mon.noname-a-legacy 2024-09-10T20:35:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:1: v1:172.21.15.176:6789/0 mon.noname-c-legacy 2024-09-10T20:35:17.890 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-b 2024-09-10T20:35:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:3: v2:172.21.15.153:3300/0 mon.noname-a 2024-09-10T20:35:17.891 INFO:teuthology.orchestra.run.smithi153.stderr:4: v2:172.21.15.176:3300/0 mon.noname-c 2024-09-10T20:35:17.891 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 1 Processor -- start 2024-09-10T20:35:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 1 -- start start 2024-09-10T20:35:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient: init 2024-09-10T20:35:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient: _reopen_session rank -1 2024-09-10T20:35:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient: _add_conns ranks=[4,3,0,2,1] 2024-09-10T20:35:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 1 --2- >> v2:172.21.15.176:3300/0 conn(0x7fda240a3eb0 0x7fda24074540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient(hunting): picked mon.noname-c con 0x7fda240a3eb0 addr v2:172.21.15.176:3300/0 2024-09-10T20:35:17.893 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fda24074a80 0x7fda241360d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient(hunting): picked mon.noname-a con 0x7fda24074a80 addr v2:172.21.15.153:3300/0 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fda24133030 addr v1:172.21.15.153:6789/0 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 1 -- --> v1:172.21.15.153:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda240a2cb0 con 0x7fda24133030 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 1 -- --> v2:172.21.15.153:3300/0 -- mon_getmap magic: 0 -- 0x7fda240a08a0 con 0x7fda24074a80 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 1 -- --> v2:172.21.15.176:3300/0 -- mon_getmap magic: 0 -- 0x7fda240a4290 con 0x7fda240a3eb0 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient(hunting): _renew_subs 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda28a00640 10 monclient(hunting): authenticate will time out at 1478.028883s 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fda24074a80 0x7fda241360d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda22d9d640 1 --1- >> v1:172.21.15.153:6789/0 conn(0x7fda24133030 0x7fda24136610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.153:6789/0 says I am v1:172.21.15.153:42680/0 (socket says 172.21.15.153:42680) 2024-09-10T20:35:17.894 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 1 --2- >> v2:172.21.15.153:3300/0 conn(0x7fda24074a80 0x7fda241360d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3300/0 says I am v2:172.21.15.153:57216/0 (socket says 172.21.15.153:57216) 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda22d9d640 1 -- 172.21.15.153:0/1587863392 learned_addr learned my addr 172.21.15.153:0/1587863392 (peer_addr_for_me v1:172.21.15.153:0/0) 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient(hunting): get_auth_request con 0x7fda24074a80 auth_method 0 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda2259c640 1 --2- 172.21.15.153:0/1587863392 >> v2:172.21.15.176:3300/0 conn(0x7fda240a3eb0 0x7fda24074540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.895 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda2259c640 10 monclient(hunting): get_auth_request con 0x7fda240a3eb0 auth_method 0 2024-09-10T20:35:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda2259c640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:35:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda2259c640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:35:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda2259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:35:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient(hunting): handle_auth_done global_id 34467 payload 274 2024-09-10T20:35:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 1 -- 172.21.15.153:0/1587863392 >> v2:172.21.15.176:3300/0 conn(0x7fda240a3eb0 msgr2=0x7fda24074540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:35:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 1 --2- 172.21.15.153:0/1587863392 >> v2:172.21.15.176:3300/0 conn(0x7fda240a3eb0 0x7fda24074540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 1 -- 172.21.15.153:0/1587863392 >> v1:172.21.15.153:6789/0 conn(0x7fda24133030 legacy=0x7fda24136610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-10T20:35:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient: _finish_hunting 0 2024-09-10T20:35:17.896 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 1 monclient: found mon.noname-a 2024-09-10T20:35:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:35:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.153:3300/0 2024-09-10T20:35:17.897 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.889+0000 7fda21d9b640 1 -- 172.21.15.153:0/1587863392 --> v2:172.21.15.153:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda240a2f90 con 0x7fda24074a80 2024-09-10T20:35:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 1 -- 172.21.15.153:0/1587863392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24074a80 msgr2=0x7fda241360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:35:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 1 --2- 172.21.15.153:0/1587863392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24074a80 0x7fda241360d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fda2409df30 tx=0x7fda1401c320 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25f85cb3116f5f5 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:35:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 1 -- 172.21.15.153:0/1587863392 <== mon.0 v2:172.21.15.153:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda140b0070 con 0x7fda24074a80 2024-09-10T20:35:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:35:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-10T20:35:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 10 monclient: dump: 2024-09-10T20:35:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:35:17.903 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 10 monclient: _finish_auth 0 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 10 monclient: _check_auth_tickets 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 1 -- 172.21.15.153:0/1587863392 <== mon.0 v2:172.21.15.153:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda140ab600 con 0x7fda24074a80 2024-09-10T20:35:17.904 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 10 monclient: handle_config config(0 keys) 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 1 -- 172.21.15.153:0/1587863392 <== mon.0 v2:172.21.15.153:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda240a0ef0 con 0x7fda24074a80 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 5 monclient: authenticate success, global_id 34467 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient: get_monmap_and_config success 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 4 set_mon_vals no callback set 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda2159a640 10 monclient: dump: 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.905 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient: shutdown 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 -- 172.21.15.153:0/1587863392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24074a80 msgr2=0x7fda241360d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 --2- 172.21.15.153:0/1587863392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24074a80 0x7fda241360d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fda2409df30 tx=0x7fda1401c320 comp rx=0 tx=0).stop 2024-09-10T20:35:17.906 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 -- 172.21.15.153:0/1587863392 shutdown_connections 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 --2- 172.21.15.153:0/1587863392 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24074a80 0x7fda241360d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 --2- 172.21.15.153:0/1587863392 >> v2:172.21.15.176:3300/0 conn(0x7fda240a3eb0 0x7fda24074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 -- 172.21.15.153:0/1587863392 >> 172.21.15.153:0/1587863392 conn(0x7fda240a7890 msgr2=0x7fda240a37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 -- 172.21.15.153:0/1587863392 shutdown_connections 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 -- 172.21.15.153:0/1587863392 wait complete. 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient: build_initial_monmap 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient: monmap: 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 0 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 0.000000 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:created 0.000000 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 0 (unknown) 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.907 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.noname-a 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.noname-b 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.noname-c 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 librados: starting msgr at 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 librados: starting objecter 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 Processor -- start 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 -- start start 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 librados: setting wanted keys 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 librados: calling monclient init 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient: init 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient: _reopen_session rank -1 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-10T20:35:17.908 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 --2- >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda24074a80 0x7fda241d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient(hunting): picked mon.noname-b con 0x7fda24074a80 addr [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda240a3eb0 0x7fda241d0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient(hunting): picked mon.noname-c con 0x7fda240a3eb0 addr [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24133030 0x7fda241d0f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient(hunting): picked mon.noname-a con 0x7fda24133030 addr [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 -- --> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] -- mon_getmap magic: 0 -- 0x7fda24140e10 con 0x7fda24133030 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 -- --> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] -- mon_getmap magic: 0 -- 0x7fda24140c90 con 0x7fda24074a80 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient(hunting): start opening mon connection 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 1 -- --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_getmap magic: 0 -- 0x7fda24140f90 con 0x7fda240a3eb0 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient(hunting): _renew_subs 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda28a00640 10 monclient(hunting): authenticate will time out at 1478.035778s 2024-09-10T20:35:17.909 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda240a3eb0 0x7fda241d0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda22d9d640 1 --2- >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24133030 0x7fda241d0f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 1 --2- >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda240a3eb0 0x7fda241d0a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.153:3301/0 says I am v2:172.21.15.153:35172/0 (socket says 172.21.15.153:35172) 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 1 -- 172.21.15.153:0/768380653 learned_addr learned my addr 172.21.15.153:0/768380653 (peer_addr_for_me v2:172.21.15.153:0/0) 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 10 monclient(hunting): get_auth_request con 0x7fda240a3eb0 auth_method 0 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda22d9d640 10 monclient(hunting): get_auth_request con 0x7fda24133030 auth_method 0 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda22d9d640 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda22d9d640 10 monclient(hunting): _init_auth method 2 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda22d9d640 10 monclient(hunting): _init_auth creating new auth 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:35:17.910 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.893+0000 7fda21d9b640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda22d9d640 10 monclient(hunting): handle_auth_reply_more payload 9 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda22d9d640 10 monclient(hunting): handle_auth_reply_more payload_len 9 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda22d9d640 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda2259c640 1 --2- 172.21.15.153:0/768380653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda24074a80 0x7fda241d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 10 monclient(hunting): handle_auth_done global_id 54209 payload 824 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 1 -- 172.21.15.153:0/768380653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda24074a80 msgr2=0x7fda241d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 1 --2- 172.21.15.153:0/768380653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda24074a80 0x7fda241d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 1 -- 172.21.15.153:0/768380653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24133030 msgr2=0x7fda241d0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 1 --2- 172.21.15.153:0/768380653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24133030 0x7fda241d0f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 10 monclient: _finish_hunting 0 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 1 monclient: found mon.noname-c 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.153:3301/0 2024-09-10T20:35:17.911 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 1 -- 172.21.15.153:0/768380653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda2409fbd0 con 0x7fda240a3eb0 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda2259c640 10 monclient: get_auth_request con 0x7fda24074a80 auth_method 0 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda2259c640 1 --2- 172.21.15.153:0/768380653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda24074a80 0x7fda241d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 1 -- 172.21.15.153:0/768380653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda240a3eb0 msgr2=0x7fda241d0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda21d9b640 1 --2- 172.21.15.153:0/768380653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda240a3eb0 0x7fda241d0a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda140ae040 tx=0x7fda140190c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 client.?.objecter ms_handle_connect 0x7fda240a3eb0 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 client.?.objecter resend_mon_ops 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 1 -- 172.21.15.153:0/768380653 <== mon.2 v2:172.21.15.153:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda140b0040 con 0x7fda240a3eb0 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 monclient: dump: 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.912 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 monclient: _finish_auth 0 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 monclient: _check_auth_tickets 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-10T20:35:17.913 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 5 monclient: authenticate success, global_id 54209 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 10 monclient: _renew_subs 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda2413d0e0 con 0x7fda240a3eb0 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 1 -- 172.21.15.153:0/768380653 <== mon.2 v2:172.21.15.153:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda140b9070 con 0x7fda240a3eb0 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 monclient: handle_config config(0 keys) 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 1 -- 172.21.15.153:0/768380653 <== mon.2 v2:172.21.15.153:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda140aaa10 con 0x7fda240a3eb0 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 monclient: dump: 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:epoch 1 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:fsid 8126fdd8-2c77-4da0-ba28-d5fdc347eec8 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:last_changed 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:created 2024-09-10T20:25:26.988641+0000 2024-09-10T20:35:17.914 INFO:teuthology.orchestra.run.smithi153.stderr:min_mon_release 19 (squid) 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:election_strategy: 1 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:0: [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] mon.a 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:1: [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] mon.b 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2: [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] mon.c 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 10 client.54209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 10 monclient: _renew_subs 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda20d99640 4 set_mon_vals no callback set 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fda2413ce00 con 0x7fda240a3eb0 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 1 librados: init done 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-10T20:35:17.915 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda24074540 con 0x7fda240a3eb0 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 1 -- 172.21.15.153:0/768380653 <== mon.2 v2:172.21.15.153:3301/0 4 ==== mgrmap(e 5) ==== 48487+0+0 (secure 0 0 0) 0x7fda140ab4c0 con 0x7fda240a3eb0 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 1 --2- 172.21.15.153:0/768380653 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda0403cea0 0x7fda0403f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 1 -- 172.21.15.153:0/768380653 <== mon.2 v2:172.21.15.153:3301/0 5 ==== osd_map(317..317 src has 206..317) ==== 4365+0+0 (secure 0 0 0) 0x7fda140e9470 con 0x7fda240a3eb0 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 10 client.54209.objecter ms_dispatch 0x7fda24000ba0 osd_map(317..317 src has 206..317) 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.897+0000 7fda12ffd640 3 client.54209.objecter handle_osd_map got epochs [317,317] > 0 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.901+0000 7fda12ffd640 3 client.54209.objecter handle_osd_map decoding full epoch 317 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.901+0000 7fda12ffd640 20 client.54209.objecter dump_active .. 0 homeless 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.901+0000 7fda2259c640 1 --2- 172.21.15.153:0/768380653 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda0403cea0 0x7fda0403f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-09-10T20:35:17.916 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.901+0000 7fda2259c640 10 monclient: get_auth_request con 0x7fda0403cea0 auth_method 0 2024-09-10T20:35:17.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.901+0000 7fda2259c640 1 --2- 172.21.15.153:0/768380653 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda0403cea0 0x7fda0403f360 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7fda18009840 tx=0x7fda18008040 comp rx=0 tx=0).ready entity=mgr.4106 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-09-10T20:35:17.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.901+0000 7fda12ffd640 10 client.54209.objecter ms_handle_connect 0x7fda0403cea0 2024-09-10T20:35:17.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.905+0000 7fda12ffd640 1 -- 172.21.15.153:0/768380653 <== mon.2 v2:172.21.15.153:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187610 (secure 0 0 0) 0x7fda14016030 con 0x7fda240a3eb0 2024-09-10T20:35:17.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.905+0000 7fda12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-10T20:35:17.917 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:17.905+0000 7fda12ffd640 10 monclient: _finish_command 1 = system:0 2024-09-10T20:35:18.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.077+0000 7fda28a00640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-10T20:35:18.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.077+0000 7fda28a00640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-10T20:35:18.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.077+0000 7fda28a00640 10 monclient: _send_mon_message to mon.c at v2:172.21.15.153:3301/0 2024-09-10T20:35:18.079 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.077+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 --> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fda2413b8e0 con 0x7fda240a3eb0 2024-09-10T20:35:18.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.085+0000 7fda12ffd640 1 -- 172.21.15.153:0/768380653 <== mon.2 v2:172.21.15.153:3301/0 7 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fda140bb6b0 con 0x7fda240a3eb0 2024-09-10T20:35:18.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.085+0000 7fda12ffd640 10 monclient: handle_config config(1 keys) 2024-09-10T20:35:18.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.085+0000 7fda20d99640 4 set_mon_vals no callback set 2024-09-10T20:35:18.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.085+0000 7fda20d99640 10 set_mon_vals mon_health_to_clog = false 2024-09-10T20:35:18.087 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.085+0000 7fda12ffd640 1 -- 172.21.15.153:0/768380653 <== mon.2 v2:172.21.15.153:3301/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v5)=0 v5) ==== 123+0+0 (secure 0 0 0) 0x7fda140be980 con 0x7fda240a3eb0 2024-09-10T20:35:18.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.085+0000 7fda12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-10T20:35:18.088 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.085+0000 7fda12ffd640 10 monclient: _finish_command 2 = system:0 2024-09-10T20:35:18.090 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 10 librados: watch_flush enter 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 10 librados: watch_flush exit 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 20 client.54209.objecter shutdown clearing up homeless session... 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 10 client.54209.objecter successfully canceled tick 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda0403cea0 msgr2=0x7fda0403f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 --2- 172.21.15.153:0/768380653 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda0403cea0 0x7fda0403f360 secure :-1 s=READY pgs=3390 cs=0 l=1 rev1=1 crypto rx=0x7fda18009840 tx=0x7fda18008040 comp rx=0 tx=0).stop 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 10 monclient: shutdown 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda240a3eb0 msgr2=0x7fda241d0a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 --2- 172.21.15.153:0/768380653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda240a3eb0 0x7fda241d0a30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fda140ae040 tx=0x7fda140190c0 comp rx=0 tx=0).stop 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 shutdown_connections 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 --2- 172.21.15.153:0/768380653 >> [v2:172.21.15.153:3300/0,v1:172.21.15.153:6789/0] conn(0x7fda24133030 0x7fda241d0f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:18.091 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 --2- 172.21.15.153:0/768380653 >> v2:172.21.15.176:6812/3003473398 conn(0x7fda0403cea0 0x7fda0403f360 unknown :-1 s=CLOSED pgs=3390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:18.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 --2- 172.21.15.153:0/768380653 >> [v2:172.21.15.153:3301/0,v1:172.21.15.153:6790/0] conn(0x7fda240a3eb0 0x7fda241d0a30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:18.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 --2- 172.21.15.153:0/768380653 >> [v2:172.21.15.176:3300/0,v1:172.21.15.176:6789/0] conn(0x7fda24074a80 0x7fda241d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-09-10T20:35:18.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 >> 172.21.15.153:0/768380653 conn(0x7fda240a7890 msgr2=0x7fda24135d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-10T20:35:18.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 shutdown_connections 2024-09-10T20:35:18.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 -- 172.21.15.153:0/768380653 wait complete. 2024-09-10T20:35:18.092 INFO:teuthology.orchestra.run.smithi153.stderr:2024-09-10T20:35:18.089+0000 7fda28a00640 1 librados: shutdown 2024-09-10T20:35:18.105 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-10T20:35:18.106 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-10T20:35:18.106 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-10T20:35:18.106 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:18.180 INFO:tasks.ceph.osd.0:Stopped 2024-09-10T20:35:18.180 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-10T20:35:18.180 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:18.265 INFO:tasks.ceph.osd.1:Stopped 2024-09-10T20:35:18.265 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-10T20:35:18.265 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:18.347 INFO:tasks.ceph.osd.2:Stopped 2024-09-10T20:35:18.347 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-10T20:35:18.347 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:18.420 INFO:tasks.ceph.osd.3:Stopped 2024-09-10T20:35:18.420 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-10T20:35:18.420 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:21.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-09-10T20:35:24.522 INFO:tasks.ceph.osd.4:Stopped 2024-09-10T20:35:24.522 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-10T20:35:24.522 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:24.596 INFO:tasks.ceph.osd.5:Stopped 2024-09-10T20:35:24.596 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-10T20:35:24.596 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-10T20:35:24.597 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:24.627 INFO:tasks.ceph.mgr.x:Stopped 2024-09-10T20:35:24.627 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-10T20:35:24.627 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-10T20:35:24.628 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:24.654 INFO:tasks.ceph.mon.a:Stopped 2024-09-10T20:35:24.655 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-10T20:35:24.655 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:24.725 INFO:tasks.ceph.mon.c:Stopped 2024-09-10T20:35:24.725 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-10T20:35:24.725 INFO:teuthology.orchestra.run:waiting for 300 2024-09-10T20:35:24.751 INFO:tasks.ceph.mon.b:Stopped 2024-09-10T20:35:24.751 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-10T20:35:24.751 DEBUG:teuthology.orchestra.run.smithi153:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-10T20:35:24.782 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-10T20:35:24.783 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-10T20:35:25.117 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-10T20:35:25.117 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-10T20:35:25.340 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi153.front.sepia.ceph.com 2024-09-10T20:35:25.340 DEBUG:teuthology.orchestra.run.smithi153:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-10T20:35:25.504 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-10T20:35:25.504 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-10T20:35:25.681 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-10T20:35:25.681 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-10T20:35:25.808 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi176.front.sepia.ceph.com 2024-09-10T20:35:25.808 DEBUG:teuthology.orchestra.run.smithi176:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-10T20:35:25.917 INFO:tasks.ceph:Archiving mon data... 2024-09-10T20:35:25.918 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561/data/mon.a.tgz 2024-09-10T20:35:25.918 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-10T20:35:25.924 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.wvLU8d2DFe 2024-09-10T20:35:25.925 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.wvLU8d2DFe 2024-09-10T20:35:26.101 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.wvLU8d2DFe 2024-09-10T20:35:26.166 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.wvLU8d2DFe is 596KB 2024-09-10T20:35:26.232 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.wvLU8d2DFe 2024-09-10T20:35:26.237 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561/data/mon.c.tgz 2024-09-10T20:35:26.237 DEBUG:teuthology.orchestra.run.smithi153:> mktemp 2024-09-10T20:35:26.244 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-09-10T20:35:26.283 INFO:teuthology.orchestra.run.smithi153.stdout:/tmp/tmp.wB9TU1Lnau 2024-09-10T20:35:26.283 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.wB9TU1Lnau 2024-09-10T20:35:26.458 DEBUG:teuthology.orchestra.run.smithi153:> sudo chmod 0666 /tmp/tmp.wB9TU1Lnau 2024-09-10T20:35:26.522 DEBUG:teuthology.orchestra.remote:smithi153:/tmp/tmp.wB9TU1Lnau is 594KB 2024-09-10T20:35:26.584 DEBUG:teuthology.orchestra.run.smithi153:> rm -fr /tmp/tmp.wB9TU1Lnau 2024-09-10T20:35:26.590 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561/data/mon.b.tgz 2024-09-10T20:35:26.591 DEBUG:teuthology.orchestra.run.smithi176:> mktemp 2024-09-10T20:35:26.597 INFO:teuthology.orchestra.run.smithi176.stdout:/tmp/tmp.lnJtoeCMvI 2024-09-10T20:35:26.598 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.lnJtoeCMvI 2024-09-10T20:35:26.775 DEBUG:teuthology.orchestra.run.smithi176:> sudo chmod 0666 /tmp/tmp.lnJtoeCMvI 2024-09-10T20:35:26.839 DEBUG:teuthology.orchestra.remote:smithi176:/tmp/tmp.lnJtoeCMvI is 597KB 2024-09-10T20:35:26.905 DEBUG:teuthology.orchestra.run.smithi176:> rm -fr /tmp/tmp.lnJtoeCMvI 2024-09-10T20:35:26.911 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-10T20:35:26.911 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-10T20:35:26.913 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-10T20:35:26.974 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-10T20:35:27.015 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-10T20:35:27.015 INFO:tasks.ceph:Archiving crash dumps... 2024-09-10T20:35:27.017 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561/remote/smithi153/crash 2024-09-10T20:35:27.017 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-10T20:35:27.031 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/lib/ceph/crash to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561/remote/smithi176/crash 2024-09-10T20:35:27.032 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-10T20:35:27.071 INFO:tasks.ceph:Compressing logs... 2024-09-10T20:35:27.071 DEBUG:teuthology.orchestra.run.smithi153:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T20:35:27.079 DEBUG:teuthology.orchestra.run.smithi176:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-10T20:35:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31434.log 2024-09-10T20:35:27.099 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30604.log 2024-09-10T20:35:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29723.log 2024-09-10T20:35:27.100 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31329.log 2024-09-10T20:35:27.100 INFO:teuthology.orchestra.run.smithi153.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31434.log.gz 2024-09-10T20:35:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28414.log 2024-09-10T20:35:27.102 INFO:teuthology.orchestra.run.smithi153.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.30604.log.gz 2024-09-10T20:35:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28090.log 2024-09-10T20:35:27.102 INFO:teuthology.orchestra.run.smithi153.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29723.log.gz 2024-09-10T20:35:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28258.log 2024-09-10T20:35:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28090.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.31329.log.gz 2024-09-10T20:35:27.102 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28409.log 2024-09-10T20:35:27.103 INFO:teuthology.orchestra.run.smithi153.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28414.log.gz 2024-09-10T20:35:27.103 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28090.log.gz 2024-09-10T20:35:27.103 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28258.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30808.log 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28255.log 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28258.log.gz 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30808.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29567.log 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28409.log.gz 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28733.log 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29567.log: gzip 87.9% -5 -- replaced with /var/log/ceph/ceph-client.admin.28255.log.gz 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30808.log.gz 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28733.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30500.log 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29567.log.gz 2024-09-10T20:35:27.106 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30887.log 2024-09-10T20:35:27.107 INFO:teuthology.orchestra.run.smithi153.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28733.log.gz 2024-09-10T20:35:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30939.log 2024-09-10T20:35:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30887.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30578.log 2024-09-10T20:35:27.107 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30500.log.gz 2024-09-10T20:35:27.107 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31121.log 2024-09-10T20:35:27.111 INFO:teuthology.orchestra.run.smithi153.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30939.log.gz 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30887.log.gz 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29395.log 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30578.log: /var/log/ceph/ceph-client.admin.31121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30860.log 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30578.log.gz 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28259.log 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31121.log.gz 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29671.log 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29395.log.gz 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30081.log 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.30860.log.gz 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30342.log 2024-09-10T20:35:27.112 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28259.log.gz 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30081.log: gzip 88.0% -- replaced with /var/log/ceph/ceph-client.admin.29671.log.gz -5 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29697.log 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28603.log 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30081.log.gz 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29697.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21480.log 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30342.log.gz 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28603.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29697.log.gz 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31225.log 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.tmp-client.admin.21480.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29775.log 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21480.log.gz 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28603.log.gz 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31225.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-10T20:35:27.113 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31173.log 2024-09-10T20:35:27.117 INFO:teuthology.orchestra.run.smithi153.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31225.log.gz 2024-09-10T20:35:27.117 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29429.log 2024-09-10T20:35:27.117 INFO:teuthology.orchestra.run.smithi153.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.29775.log.gz 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28411.log 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29429.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29922.log 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31173.log.gz 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28863.log 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29429.log.gz 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29922.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28411.log.gz 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30708.log 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2%/var/log/ceph/ceph-client.admin.28863.log: -- replaced with /var/log/ceph/ceph-client.admin.29922.log.gz 2024-09-10T20:35:27.118 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30107.log 2024-09-10T20:35:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29948.log 2024-09-10T20:35:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30708.log: /var/log/ceph/ceph-client.admin.30107.log: 90.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29749.log 2024-09-10T20:35:27.119 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28863.log.gz 2024-09-10T20:35:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28655.log 2024-09-10T20:35:27.119 INFO:teuthology.orchestra.run.smithi153.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30708.log.gz 2024-09-10T20:35:27.119 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30107.log.gz 2024-09-10T20:35:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30526.log 2024-09-10T20:35:27.119 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30630.log 2024-09-10T20:35:27.119 INFO:teuthology.orchestra.run.smithi153.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29948.log.gz 2024-09-10T20:35:27.123 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28257.log 2024-09-10T20:35:27.123 INFO:teuthology.orchestra.run.smithi153.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28655.log.gz 2024-09-10T20:35:27.123 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30630.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.29749.log.gz 2024-09-10T20:35:27.123 INFO:teuthology.orchestra.run.smithi153.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.30526.log.gz 2024-09-10T20:35:27.123 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29365.log 2024-09-10T20:35:27.123 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29976.log 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28257.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30630.log.gz 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29365.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28244.log 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29976.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28257.log.gz 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29483.log 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28244.log: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.29365.log.gz 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29593.log 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29976.log.gz 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30368.log 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28244.log.gz 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30991.log 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.29483.log.gz 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29593.log.gz 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30189.log 2024-09-10T20:35:27.124 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30474.log 2024-09-10T20:35:27.125 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30991.log: /var/log/ceph/ceph-client.admin.30189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31410.log 2024-09-10T20:35:27.125 INFO:teuthology.orchestra.run.smithi153.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.30368.log.gz 2024-09-10T20:35:27.125 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30991.log.gz 2024-09-10T20:35:27.125 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28551.log 2024-09-10T20:35:27.125 INFO:teuthology.orchestra.run.smithi153.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.30189.log.gz 2024-09-10T20:35:27.125 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25264.log 2024-09-10T20:35:27.125 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30474.log.gz 2024-09-10T20:35:27.125 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28629.log 2024-09-10T20:35:27.125 INFO:teuthology.orchestra.run.smithi153.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31410.log.gz 2024-09-10T20:35:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.25264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29830.log 2024-09-10T20:35:27.130 INFO:teuthology.orchestra.run.smithi153.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28551.log.gz 2024-09-10T20:35:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31017.log 2024-09-10T20:35:27.130 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.25264.log.gz 2024-09-10T20:35:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29830.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-10T20:35:27.130 INFO:teuthology.orchestra.run.smithi153.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.28629.log.gz 2024-09-10T20:35:27.130 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25078.log 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29830.log.gz 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28577.log 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31017.log.gz 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.25078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29457.log 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30656.log 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28577.log: 91.6%/var/log/ceph/ceph-client.admin.29457.log: -- replaced with /var/log/ceph/ceph-client.admin.25078.log.gz 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30834.log 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28577.log.gz 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30656.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.29457.log.gz 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr: 90.5% -- replaced with /var/log/ceph/ceph-client.admin.30656.log.gz 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25195.log 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31095.log 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.25195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30743.log 2024-09-10T20:35:27.131 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.30834.log.gz 2024-09-10T20:35:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28759.log 2024-09-10T20:35:27.132 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30316.log 2024-09-10T20:35:27.132 INFO:teuthology.orchestra.run.smithi153.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.25195.log.gz 2024-09-10T20:35:27.132 INFO:teuthology.orchestra.run.smithi153.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.31095.log.gz 2024-09-10T20:35:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28759.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30743.log.gz 2024-09-10T20:35:27.134 INFO:teuthology.orchestra.run.smithi153.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28759.log.gz 2024-09-10T20:35:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31355.log 2024-09-10T20:35:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30682.log 2024-09-10T20:35:27.134 INFO:teuthology.orchestra.run.smithi153.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.30316.log.gz 2024-09-10T20:35:27.134 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28187.log 2024-09-10T20:35:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30682.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29895.log 2024-09-10T20:35:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28187.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31355.log.gz 2024-09-10T20:35:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31147.log 2024-09-10T20:35:27.135 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30682.log.gz 2024-09-10T20:35:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29895.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28187.log.gz 2024-09-10T20:35:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28707.log 2024-09-10T20:35:27.135 INFO:teuthology.orchestra.run.smithi153.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.29895.log.gz 2024-09-10T20:35:27.135 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31147.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28707.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30394.log 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31147.log.gz 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28707.log.gz -5 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29619.log 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29535.log 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30133.log 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3%/var/log/ceph/ceph-client.admin.29535.log: -- replaced with /var/log/ceph/ceph-client.admin.30394.log.gz 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.29619.log.gz 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29535.log.gz 2024-09-10T20:35:27.137 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30420.log 2024-09-10T20:35:27.141 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31069.log 2024-09-10T20:35:27.141 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30420.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28400.log 2024-09-10T20:35:27.141 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30133.log.gz 2024-09-10T20:35:27.141 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31069.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30420.log.gz 2024-09-10T20:35:27.141 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.31069.log.gz 2024-09-10T20:35:27.142 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29339.log 2024-09-10T20:35:27.142 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31303.log 2024-09-10T20:35:27.142 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29339.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-10T20:35:27.142 INFO:teuthology.orchestra.run.smithi153.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28400.log.gz 2024-09-10T20:35:27.142 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31303.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29339.log.gz 2024-09-10T20:35:27.142 INFO:teuthology.orchestra.run.smithi153.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31303.log.gz 2024-09-10T20:35:27.142 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30552.log 2024-09-10T20:35:27.144 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29509.log 2024-09-10T20:35:27.144 INFO:teuthology.orchestra.run.smithi153.stderr: 91.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-10T20:35:27.144 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28213.log 2024-09-10T20:35:27.144 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31251.log 2024-09-10T20:35:27.144 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5%/var/log/ceph/ceph-client.admin.29509.log: -- replaced with /var/log/ceph/ceph-client.admin.30552.log.gz 2024-09-10T20:35:27.144 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28213.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.29509.log.gz 2024-09-10T20:35:27.144 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28213.log.gz 2024-09-10T20:35:27.144 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30448.log 2024-09-10T20:35:27.145 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28135.log 2024-09-10T20:35:27.145 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31199.log 2024-09-10T20:35:27.145 INFO:teuthology.orchestra.run.smithi153.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31251.log.gz 2024-09-10T20:35:27.145 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28135.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30448.log.gz 2024-09-10T20:35:27.146 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29111.log 2024-09-10T20:35:27.146 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28614.log 2024-09-10T20:35:27.147 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28115.log 2024-09-10T20:35:27.148 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28135.log.gz 2024-09-10T20:35:27.148 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28915.log 2024-09-10T20:35:27.148 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28115.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-10T20:35:27.148 INFO:teuthology.orchestra.run.smithi153.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31199.log.gz 2024-09-10T20:35:27.148 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28915.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28115.log.gz 2024-09-10T20:35:27.148 INFO:teuthology.orchestra.run.smithi153.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28915.log.gz 2024-09-10T20:35:27.152 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28415.log 2024-09-10T20:35:27.152 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29645.log 2024-09-10T20:35:27.152 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28415.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29801.log 2024-09-10T20:35:27.153 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29645.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.28415.log.gz 2024-09-10T20:35:27.153 INFO:teuthology.orchestra.run.smithi153.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.29645.log.gz 2024-09-10T20:35:27.153 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30965.log 2024-09-10T20:35:27.153 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.29801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31381.log 2024-09-10T20:35:27.153 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30965.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29801.log.gz 2024-09-10T20:35:27.153 INFO:teuthology.orchestra.run.smithi153.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30965.log.gz 2024-09-10T20:35:27.153 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28837.log 2024-09-10T20:35:27.161 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29112.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28614.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28794.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29116.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28794.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28614.log.gz 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27973.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28639.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29116.log: /var/log/ceph/ceph-client.admin.27973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28531.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29111.log.gz 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29113.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:gzip/var/log/ceph/ceph-client.admin.28531.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28317.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29113.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.29112.log.gz 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27978.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28470.log 2024-09-10T20:35:27.162 INFO:teuthology.orchestra.run.smithi176.stderr: 91.5%/var/log/ceph/ceph-client.admin.27978.log: -- replaced with /var/log/ceph/ceph-client.admin.29116.log.gz 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28469.log 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28470.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28794.log.gz 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29113.log.gz 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.28470.log.gz 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28096.log 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29115.log 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28096.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28888.log 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29115.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29115.log.gz 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.28531.log.gz 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.27978.log.gz 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28388.log 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28888.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28473.log 2024-09-10T20:35:27.163 INFO:teuthology.orchestra.run.smithi176.stderr: 90.8%/var/log/ceph/ceph-client.admin.28388.log: -- replaced with /var/log/ceph/ceph-client.admin.28469.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28421.log 2024-09-10T20:35:27.164 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:35:27.164 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28735.log 2024-09-10T20:35:27.164 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28421.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28421.log.gz 88.8% -- replaced with /var/log/ceph/ceph-client.admin.28473.log.gz 2024-09-10T20:35:27.164 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:35:27.164 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28028.log 2024-09-10T20:35:27.164 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25181.log 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30028.log 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31381.log: /var/log/ceph/ceph-client.admin.28837.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31381.log.gz 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28837.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28161.log 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30028.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.30028.log.gz 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28681.log 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30055.log 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28161.log: 93.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28785.log 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30055.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.28681.log.gz 2024-09-10T20:35:27.167 INFO:teuthology.orchestra.run.smithi153.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28161.log.gz 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28889.log 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30055.log.gz 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28407.log 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28785.log: /var/log/ceph/ceph-client.admin.28889.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31043.log 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28407.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.28889.log.gz 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.28785.log.gz 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28407.log.gz 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30913.log 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30002.log 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31043.log: /var/log/ceph/ceph-client.admin.30913.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28811.log 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30002.log: 87.8% 88.2% -- replaced with /var/log/ceph/ceph-client.admin.31043.log.gz 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30913.log.gz 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28065.log 2024-09-10T20:35:27.168 INFO:teuthology.orchestra.run.smithi153.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.30002.log.gz 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28811.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28250.log 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28065.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30215.log 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.28811.log.gz 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.28250.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.28065.log.gz 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28250.log.gz 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27997.log 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31277.log 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.27997.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30161.log 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.30215.log.gz 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.31277.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.27997.log.gz 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31277.log.gz 2024-09-10T20:35:27.169 INFO:teuthology.orchestra.run.smithi153.stderr:/var/log/ceph/ceph-client.admin.30161.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.30161.log.gz 2024-09-10T20:35:27.172 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28028.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28317.log.gz 2024-09-10T20:35:27.172 INFO:teuthology.orchestra.run.smithi176.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.28639.log.gz 2024-09-10T20:35:27.172 INFO:teuthology.orchestra.run.smithi176.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28096.log.gz 2024-09-10T20:35:27.172 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25157.log 2024-09-10T20:35:27.172 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.25181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28060.log 2024-09-10T20:35:27.172 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.25157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28702.log 2024-09-10T20:35:27.172 INFO:teuthology.orchestra.run.smithi176.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25181.log.gz 2024-09-10T20:35:27.173 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28060.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25157.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28218.log 2024-09-10T20:35:27.173 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:35:27.173 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27941.log 2024-09-10T20:35:27.173 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28702.log: /var/log/ceph/ceph-client.admin.28218.log: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.28735.log.gz 2024-09-10T20:35:27.173 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25133.log 2024-09-10T20:35:27.176 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27941.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28218.log.gz 2024-09-10T20:35:27.176 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-10T20:35:27.177 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.25133.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.25133.log.gz 2024-09-10T20:35:27.178 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-10T20:35:27.191 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28273.log 2024-09-10T20:35:27.204 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.5.log: 92.8% -- replaced with /var/log/ceph/ceph-client.admin.28888.log.gz 2024-09-10T20:35:27.207 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28412.log 2024-09-10T20:35:27.209 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28273.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.28388.log.gz 2024-09-10T20:35:27.211 INFO:teuthology.orchestra.run.smithi176.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.27973.log.gz 2024-09-10T20:35:27.211 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29061.log 2024-09-10T20:35:27.213 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28412.log: 92.9% -- replaced with /var/log/ceph/ceph-client.admin.28702.log.gz 2024-09-10T20:35:27.214 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28165.log 2024-09-10T20:35:27.214 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29061.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.28273.log.gz 2024-09-10T20:35:27.215 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29441.log 2024-09-10T20:35:27.216 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28165.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.29061.log.gz 2024-09-10T20:35:27.217 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28705.log 2024-09-10T20:35:27.218 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29441.log: 93.9% -- replaced with /var/log/ceph/ceph-client.admin.28412.log.gz 2024-09-10T20:35:27.218 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29566.log 2024-09-10T20:35:27.219 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28705.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28165.log.gz 2024-09-10T20:35:27.219 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28663.log 2024-09-10T20:35:27.220 INFO:teuthology.orchestra.run.smithi176.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29441.log.gz 2024-09-10T20:35:27.220 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28404.log 2024-09-10T20:35:27.222 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29566.log: /var/log/ceph/ceph-client.admin.28663.log: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28663.log.gz 2024-09-10T20:35:27.223 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25109.log 2024-09-10T20:35:27.223 INFO:teuthology.orchestra.run.smithi176.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29566.log.gz 2024-09-10T20:35:27.223 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27991.log 2024-09-10T20:35:27.225 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.25109.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.25109.log.gz 2024-09-10T20:35:27.225 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29443.log 2024-09-10T20:35:27.226 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27991.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.28404.log.gz 2024-09-10T20:35:27.226 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27949.log 2024-09-10T20:35:27.229 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29443.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.27991.log.gz 2024-09-10T20:35:27.230 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29133.log 2024-09-10T20:35:27.230 INFO:teuthology.orchestra.run.smithi176.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.29443.log.gz 2024-09-10T20:35:27.230 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-10T20:35:27.231 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27949.log: /var/log/ceph/ceph-client.admin.29133.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.27941.log.gz 2024-09-10T20:35:27.232 INFO:teuthology.orchestra.run.smithi176.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.29133.log.gz 2024-09-10T20:35:27.232 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-10T20:35:27.232 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28249.log 2024-09-10T20:35:27.237 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.log: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.27949.log.gz 2024-09-10T20:35:27.237 INFO:teuthology.orchestra.run.smithi176.stderr: 86.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-10T20:35:27.237 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28462.log 2024-09-10T20:35:27.238 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28295.log 2024-09-10T20:35:27.240 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28462.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.28462.log.gz 2024-09-10T20:35:27.240 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29440.log 2024-09-10T20:35:27.243 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28295.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.28705.log.gz 2024-09-10T20:35:27.244 INFO:teuthology.orchestra.run.smithi176.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.28249.log.gz 2024-09-10T20:35:27.246 INFO:teuthology.orchestra.run.smithi176.stderr: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.28295.log.gz 2024-09-10T20:35:27.247 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28135.log 2024-09-10T20:35:27.247 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29062.log 2024-09-10T20:35:27.248 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28869.log 2024-09-10T20:35:27.250 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29062.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29440.log.gz 2024-09-10T20:35:27.250 INFO:teuthology.orchestra.run.smithi176.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.29062.log.gz 2024-09-10T20:35:27.251 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28111.log 2024-09-10T20:35:27.251 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29565.log 2024-09-10T20:35:27.253 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28869.log: /var/log/ceph/ceph-client.admin.28111.log: 95.2% -- replaced with /var/log/ceph/ceph-client.admin.28060.log.gz 2024-09-10T20:35:27.253 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27956.log 2024-09-10T20:35:27.253 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29565.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28869.log.gz 2024-09-10T20:35:27.254 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-10T20:35:27.256 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27956.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29565.log.gz 2024-09-10T20:35:27.256 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29132.log 2024-09-10T20:35:27.258 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph.audit.log: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28111.log.gz 2024-09-10T20:35:27.258 INFO:teuthology.orchestra.run.smithi176.stderr: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.28135.log.gz 2024-09-10T20:35:27.259 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29564.log 2024-09-10T20:35:27.259 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29132.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28463.log 2024-09-10T20:35:27.262 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29564.log: 91.9% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-10T20:35:27.262 INFO:teuthology.orchestra.run.smithi176.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29132.log.gz 2024-09-10T20:35:27.263 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28461.log 2024-09-10T20:35:27.263 INFO:teuthology.orchestra.run.smithi176.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29564.log.gz 2024-09-10T20:35:27.263 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27932.log 2024-09-10T20:35:27.263 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28463.log: /var/log/ceph/ceph-client.admin.28461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28816.log 2024-09-10T20:35:27.264 INFO:teuthology.orchestra.run.smithi153.stderr: 91.2% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-10T20:35:27.266 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27932.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.28463.log.gz 2024-09-10T20:35:27.266 INFO:teuthology.orchestra.run.smithi176.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.28461.log.gz 2024-09-10T20:35:27.266 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28837.log 2024-09-10T20:35:27.267 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29114.log 2024-09-10T20:35:27.270 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28816.log: /var/log/ceph/ceph-client.admin.28837.log: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.27956.log.gz 2024-09-10T20:35:27.270 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28464.log 2024-09-10T20:35:27.274 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29114.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29114.log.gz 2024-09-10T20:35:27.274 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25205.log 2024-09-10T20:35:27.275 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28464.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28464.log.gz 2024-09-10T20:35:27.276 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-10T20:35:27.277 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.25205.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.25205.log.gz 2024-09-10T20:35:27.277 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29442.log 2024-09-10T20:35:27.277 INFO:teuthology.orchestra.run.smithi176.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.28837.log.gz 2024-09-10T20:35:27.279 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-10T20:35:27.280 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29442.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28816.log.gz 2024-09-10T20:35:27.280 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.27965.log 2024-09-10T20:35:27.282 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-osd.4.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.29442.log.gz 2024-09-10T20:35:27.287 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29444.log 2024-09-10T20:35:27.289 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.27965.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.27965.log.gz 2024-09-10T20:35:27.289 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28205.log 2024-09-10T20:35:27.293 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29444.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29444.log.gz 2024-09-10T20:35:27.293 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29060.log 2024-09-10T20:35:27.296 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.28205.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28205.log.gz 2024-09-10T20:35:27.297 INFO:teuthology.orchestra.run.smithi176.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25229.log 2024-09-10T20:35:27.298 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.29060.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.27932.log.gz 2024-09-10T20:35:27.300 INFO:teuthology.orchestra.run.smithi176.stderr:/var/log/ceph/ceph-client.admin.25229.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.25229.log.gz 2024-09-10T20:35:27.300 INFO:teuthology.orchestra.run.smithi176.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29060.log.gz 2024-09-10T20:35:27.460 INFO:teuthology.orchestra.run.smithi176.stderr: 92.0% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-10T20:35:28.227 INFO:teuthology.orchestra.run.smithi153.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-10T20:35:28.366 INFO:teuthology.orchestra.run.smithi176.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28028.log.gz 2024-09-10T20:35:28.524 INFO:teuthology.orchestra.run.smithi176.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-10T20:35:31.121 INFO:teuthology.orchestra.run.smithi176.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-10T20:35:31.205 INFO:teuthology.orchestra.run.smithi176.stderr: 92.3% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-10T20:35:31.339 INFO:teuthology.orchestra.run.smithi176.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-10T20:35:31.341 INFO:teuthology.orchestra.run.smithi176.stderr: 2024-09-10T20:35:31.341 INFO:teuthology.orchestra.run.smithi176.stderr:real 0m4.217s 2024-09-10T20:35:31.341 INFO:teuthology.orchestra.run.smithi176.stderr:user 0m14.852s 2024-09-10T20:35:31.341 INFO:teuthology.orchestra.run.smithi176.stderr:sys 0m0.569s 2024-09-10T20:35:31.437 INFO:teuthology.orchestra.run.smithi153.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-10T20:35:31.764 INFO:teuthology.orchestra.run.smithi153.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-10T20:35:31.802 INFO:teuthology.orchestra.run.smithi153.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-10T20:35:31.804 INFO:teuthology.orchestra.run.smithi153.stderr: 2024-09-10T20:35:31.804 INFO:teuthology.orchestra.run.smithi153.stderr:real 0m4.723s 2024-09-10T20:35:31.804 INFO:teuthology.orchestra.run.smithi153.stderr:user 0m14.590s 2024-09-10T20:35:31.804 INFO:teuthology.orchestra.run.smithi153.stderr:sys 0m0.525s 2024-09-10T20:35:31.804 INFO:tasks.ceph:Archiving logs... 2024-09-10T20:35:31.805 DEBUG:teuthology.misc:Transferring archived files from smithi153:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561/remote/smithi153/log 2024-09-10T20:35:31.806 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T20:35:33.960 DEBUG:teuthology.misc:Transferring archived files from smithi176:/var/log/ceph to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561/remote/smithi176/log 2024-09-10T20:35:33.960 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-10T20:35:35.945 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-10T20:35:35.954 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-10T20:35:35.955 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T20:35:35.957 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-10T20:35:36.039 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-10T20:35:36.047 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-10T20:35:36.047 DEBUG:teuthology.orchestra.run.smithi153:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:35:36.050 DEBUG:teuthology.orchestra.run.smithi176:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-10T20:35:36.061 INFO:teuthology.orchestra.run.smithi176.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T20:35:36.061 INFO:teuthology.orchestra.run.smithi176.stdout:============================================================================== 2024-09-10T20:35:36.061 INFO:teuthology.orchestra.run.smithi176.stdout:*hv01.front.sepi 67.205.162.81 3 u 57 64 377 0.085 -1.024 2.069 2024-09-10T20:35:36.061 INFO:teuthology.orchestra.run.smithi176.stdout:+hv02.front.sepi 63.231.80.2 3 u 63 64 377 0.072 -1.737 0.899 2024-09-10T20:35:36.061 INFO:teuthology.orchestra.run.smithi176.stdout:+hv03.front.sepi 74.6.168.72 3 u 58 64 377 0.091 +0.382 1.663 2024-09-10T20:35:36.061 INFO:teuthology.orchestra.run.smithi176.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2024-09-10T20:35:36.062 INFO:teuthology.orchestra.run.smithi153.stdout: remote refid st t when poll reach delay offset jitter 2024-09-10T20:35:36.063 INFO:teuthology.orchestra.run.smithi153.stdout:============================================================================== 2024-09-10T20:35:36.063 INFO:teuthology.orchestra.run.smithi153.stdout:*hv01.front.sepi 67.205.162.81 3 u 61 64 377 0.090 +0.014 1.272 2024-09-10T20:35:36.063 INFO:teuthology.orchestra.run.smithi153.stdout:+hv02.front.sepi 63.231.80.2 3 u 59 64 377 0.074 -3.799 0.733 2024-09-10T20:35:36.063 INFO:teuthology.orchestra.run.smithi153.stdout:+hv03.front.sepi 74.6.168.72 3 u 59 64 377 0.089 -0.708 1.093 2024-09-10T20:35:36.063 INFO:teuthology.orchestra.run.smithi153.stdout: hv04.front.sepi .INIT. 16 u - 128 0 0.000 +0.000 0.000 2024-09-10T20:35:36.063 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-10T20:35:36.072 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-10T20:35:36.072 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-10T20:35:36.080 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-10T20:35:36.087 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-10T20:35:36.098 INFO:teuthology.task.internal:Duration was 1174.003580 seconds 2024-09-10T20:35:36.099 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-10T20:35:36.232 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-10T20:35:36.232 DEBUG:teuthology.orchestra.run.smithi153:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T20:35:36.234 DEBUG:teuthology.orchestra.run.smithi176:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-10T20:35:36.272 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-10T20:35:36.272 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi153.front.sepia.ceph.com 2024-09-10T20:35:36.272 DEBUG:teuthology.orchestra.run.smithi153:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T20:35:36.324 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi176.front.sepia.ceph.com 2024-09-10T20:35:36.324 DEBUG:teuthology.orchestra.run.smithi176:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-10T20:35:36.335 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-10T20:35:36.335 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T20:35:36.370 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-10T20:35:36.409 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-10T20:35:36.409 DEBUG:teuthology.orchestra.run.smithi153:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T20:35:36.439 DEBUG:teuthology.orchestra.run.smithi176:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-10T20:35:36.507 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-10T20:35:36.517 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-10T20:35:36.517 DEBUG:teuthology.orchestra.run.smithi153:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T20:35:36.527 DEBUG:teuthology.orchestra.run.smithi176:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-10T20:35:36.540 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-10T20:35:36.550 DEBUG:teuthology.orchestra.run.smithi153:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:35:36.571 DEBUG:teuthology.orchestra.run.smithi176:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:35:36.580 INFO:teuthology.orchestra.run.smithi153.stdout:kernel.core_pattern = core 2024-09-10T20:35:36.593 INFO:teuthology.orchestra.run.smithi176.stdout:kernel.core_pattern = core 2024-09-10T20:35:36.611 DEBUG:teuthology.orchestra.run.smithi153:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:35:36.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:35:36.662 DEBUG:teuthology.orchestra.run.smithi176:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-10T20:35:36.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-10T20:35:36.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-10T20:35:36.676 INFO:teuthology.task.internal:Transferring archived files... 2024-09-10T20:35:36.677 DEBUG:teuthology.misc:Transferring archived files from smithi153:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561/remote/smithi153 2024-09-10T20:35:36.678 DEBUG:teuthology.orchestra.run.smithi153:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T20:35:36.726 DEBUG:teuthology.misc:Transferring archived files from smithi176:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/skanta-2024-09-10_15:30:27-rados-wip-bharath7-testing-2024-09-10-1409-distro-default-smithi/7899561/remote/smithi176 2024-09-10T20:35:36.727 DEBUG:teuthology.orchestra.run.smithi176:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-10T20:35:36.765 INFO:teuthology.task.internal:Removing archive directory... 2024-09-10T20:35:36.766 DEBUG:teuthology.orchestra.run.smithi153:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T20:35:36.768 DEBUG:teuthology.orchestra.run.smithi176:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-10T20:35:36.792 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-10T20:35:36.805 INFO:teuthology.task.internal:Not uploading archives. 2024-09-10T20:35:36.806 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-10T20:35:36.814 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-10T20:35:36.814 DEBUG:teuthology.orchestra.run.smithi153:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T20:35:36.817 DEBUG:teuthology.orchestra.run.smithi176:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-10T20:35:36.820 INFO:teuthology.orchestra.run.smithi153.stdout: 658639 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 20:35 /home/ubuntu/cephtest 2024-09-10T20:35:36.839 INFO:teuthology.orchestra.run.smithi176.stdout: 658622 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 10 20:35 /home/ubuntu/cephtest 2024-09-10T20:35:36.840 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-10T20:35:36.850 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-10T20:35:36.920 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/{bluestore-options/write$/{write_random} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/force-sync-many workloads/rados_api_tests} duration: 1174.0035800933838 flavor: default owner: scheduled_skanta@teuthology success: true 2024-09-10T20:35:36.920 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-10T20:35:37.003 INFO:teuthology.run:pass